meerschaum

Meerschaum banner

Meerschaum Python API

Welcome to the Meerschaum Python API technical documentation! Here you can find information about the classes and functions provided by the meerschaum package. Visit meerschaum.io for general usage documentation.

Root Module

For your convenience, the following classes and functions may be imported from the root meerschaum namespace:

Classes

Functions

Examples

Build a Connector

Get existing connectors or build a new one in-memory with the meerschaum.get_connector() factory function:

import meerschaum as mrsm

sql_conn = mrsm.get_connector(
    'sql:temp',
    flavor='sqlite',
    database='/tmp/tmp.db',
)
df = sql_conn.read("SELECT 1 AS foo")
print(df)
# shape: (1, 1)
# ┌─────┐
# │ foo │
# │ --- │
# │ i64 │
# ╞═════╡
# │ 1   │
# └─────┘

sql_conn.to_sql(df, 'foo')
print(sql_conn.read('foo'))
#    foo
# 0    1

Create a Custom Connector Class

Decorate your connector classes with meerschaum.make_connector() to designate it as a custom connector:

from datetime import datetime, timezone
from random import randint
import meerschaum as mrsm
from meerschaum.utils.dtypes import round_time

@mrsm.make_connector
class FooConnector(mrsm.Connector):
    REQUIRED_ATTRIBUTES = ['username', 'password']

    def fetch(
        self,
        begin: datetime | None = None,
        end: datetime | None = None,
    ):
        now = begin or round_time(datetime.now(timezone.utc))
        return [
            {'ts': now, 'id': 1, 'vl': randint(1, 100)},
            {'ts': now, 'id': 2, 'vl': randint(1, 100)},
            {'ts': now, 'id': 3, 'vl': randint(1, 100)},
        ]

foo_conn = mrsm.get_connector(
    'foo:bar',
    username='foo',
    password='bar',
)
docs = foo_conn.fetch()

Build a Pipe

Build a meerschaum.Pipe in-memory:

from datetime import datetime
import meerschaum as mrsm

pipe = mrsm.Pipe(
    foo_conn, 'demo',
    instance=sql_conn,
    columns={'datetime': 'ts', 'id': 'id'},
    tags=['production'],
)
pipe.sync(begin=datetime(2024, 1, 1))
df = pipe.get_data()
print(df)
# shape: (3, 3)
# ┌─────────────────────┬─────┬─────┐
# │ ts                  ┆ id  ┆ vl  │
# │ ---                 ┆ --- ┆ --- │
# │ datetime[μs]        ┆ i64 ┆ i64 │
# ╞═════════════════════╪═════╪═════╡
# │ 2024-01-01 00:00:00 ┆ 1   ┆ 97  │
# │ 2024-01-01 00:00:00 ┆ 2   ┆ 18  │
# │ 2024-01-01 00:00:00 ┆ 3   ┆ 96  │
# └─────────────────────┴─────┴─────┘

DataFrames are Polars by default (pip install meerschaum[polars]); set dataframe: pandas in meerschaum.yaml (a root created before 5.0.0rc15 has it) or pass as_pandas=True for pandas.

Add temporary=True to skip registering the pipe in the pipes table.

Query an Integer-Axis Pipe by Datetime

When a pipe's datetime axis is an integer epoch, set precision so datetime bounds can be translated to the axis's integer value. This lets you pass a datetime begin / end to meerschaum.Pipe.get_data() (and to actions like show data, clear, and deduplicate):

from datetime import datetime, timezone
import meerschaum as mrsm

pipe = mrsm.Pipe(
    'demo', 'epoch',
    instance='sql:temp',
    columns={'datetime': 'ts'},
    dtypes={'ts': 'int'},
    precision='millisecond',
)
pipe.sync([{'ts': 1780099200000}])

### The datetime is translated to the epoch value `1780099200000`.
df = pipe.get_data(begin='2026-05-30')

Integer bounds (begin=1780099200000) still pass through unchanged. A datetime bound on a non-epoch integer axis (no precision set) raises a ValueError. Convert directly with meerschaum.utils.dtypes.datetime_to_int():

from datetime import datetime, timezone
from meerschaum.utils.dtypes import datetime_to_int

datetime_to_int(datetime(2026, 5, 30, tzinfo=timezone.utc), 'millisecond')
# 1780099200000

Get Registered Pipes

The meerschaum.get_pipes() function returns a dictionary hierarchy of pipes by connector, metric, and location:

import meerschaum as mrsm

pipes = mrsm.get_pipes(instance='sql:temp')
pipe = pipes['foo:bar']['demo'][None]

Add as_list=True to flatten the hierarchy:

import meerschaum as mrsm

pipes = mrsm.get_pipes(
    tags=['production'],
    instance=sql_conn,
    as_list=True,
)
print(pipes)
# [Pipe('foo:bar', 'demo', instance='sql:temp')]

Filter by the dtype of the datetime index column with datetime_dtypes. Accepted values are 'datetime', 'int', and 'None'; prefix with '_' to negate:

import meerschaum as mrsm

### Only pipes with a timestamp datetime index:
timestamp_pipes = mrsm.get_pipes(datetime_dtypes=['datetime'], as_list=True)

### Only pipes with an integer datetime index:
int_pipes = mrsm.get_pipes(datetime_dtypes=['int'], as_list=True)

### Exclude pipes without a datetime index:
datetime_pipes = mrsm.get_pipes(datetime_dtypes=['_None'], as_list=True)

Sync Pipes in Dependency Order with a Pipeline

A meerschaum.Pipeline syncs a set of pipes in dependency order through the Syncing Engine. Plan it first (a dry run), sync it, and save it under a name to load it again later:

import meerschaum as mrsm

pipeline = mrsm.Pipeline(pipe, parents=True)   # this pipe and every pipe it reads
print(pipeline.plan())                         # what would sync, and why
run = pipeline.sync()
print(run.success, run.inserted, run.updated)

success, msg = pipeline.save('demo')
pipeline = mrsm.Pipeline('demo', instance='sql:temp')
print(mrsm.get_pipelines(instance='sql:temp'))
# [Pipeline('demo', instance='sql:temp')]
print(mrsm.get_runs(instance='sql:temp', pipeline='demo', limit=1))

Import Plugins

You can import a plugin's module through meerschaum.Plugin.module:

import meerschaum as mrsm

plugin = mrsm.Plugin('noaa')
with mrsm.Venv(plugin):
    noaa = plugin.module

If your plugin has submodules, use meerschaum.plugins.from_plugin_import:

from meerschaum.plugins import from_plugin_import
get_defined_pipes = from_plugin_import('compose.utils.pipes', 'get_defined_pipes')

Import multiple plugins with meerschaum.plugins.import_plugins:

from meerschaum.plugins import import_plugins
noaa, compose = import_plugins('noaa', 'compose')

Create a Job

Create a meerschaum.Job with name and sysargs:

import meerschaum as mrsm

job = mrsm.Job('syncing-engine', 'sync pipes --loop')
success, msg = job.start()

Pass executor_keys as the connectors keys of an API instance to create a remote job:

import meerschaum as mrsm

job = mrsm.Job(
    'foo',
    'sync pipes -s daily',
    executor_keys='api:main',
)

Import from a Virtual Environment Use the meerschaum.Venv context manager to activate a virtual environment:

import meerschaum as mrsm

with mrsm.Venv('noaa'):
    import requests

print(requests.__file__)
# /home/bmeares/.config/meerschaum/venvs/noaa/lib/python3.12/site-packages/requests/__init__.py

To import packages which may not be installed, use meerschaum.attempt_import():

import meerschaum as mrsm

requests = mrsm.attempt_import('requests', venv='noaa')
print(requests.__file__)
# /home/bmeares/.config/meerschaum/venvs/noaa/lib/python3.12/site-packages/requests/__init__.py

Run Actions

Run sysargs with meerschaum.entry():

import meerschaum as mrsm

success, msg = mrsm.entry('show pipes + show version : x2')

Use meerschaum.actions.get_action() to access an action function directly:

from meerschaum.actions import get_action

show_pipes = get_action(['show', 'pipes'])
success, msg = show_pipes(connector_keys=['plugin:noaa'])

Get a dictionary of available subactions with meerschaum.actions.get_subactions():

from meerschaum.actions import get_subactions

subactions = get_subactions('show')
success, msg = subactions['pipes']()

Create a Plugin

Run bootstrap plugin to create a new plugin:

mrsm bootstrap plugin example

This will create example.py in your plugins directory (default ~/.config/meerschaum/plugins/, Windows: %APPDATA%\Meerschaum\plugins). You may paste the example code from the "Create a Custom Action" example below.

Open your plugin with edit plugin:

mrsm edit plugin example

Run edit plugin and paste the example code below to try out the features.

See the writing plugins guide for more in-depth documentation.

Create a Custom Action

Decorate a function with meerschaum.actions.make_action to designate it as an action. Subactions will be automatically detected if not decorated:

from meerschaum.actions import make_action

@make_action
def sing():
    print('What would you like me to sing?')
    return True, "Success"

def sing_tune():
    return False, "I don't know that song!"

def sing_song():
    print('Hello, World!')
    return True, "Success"

Use meerschaum.plugins.add_plugin_argument() to create new parameters for your action:

from meerschaum.plugins import make_action, add_plugin_argument

add_plugin_argument(
    '--song', type=str, help='What song to sing.',
)

@make_action
def sing_melody(action=None, song=None):
    to_sing = action[0] if action else song
    if not to_sing:
        return False, "Please tell me what to sing!"

    return True, f'~I am singing {to_sing}~'

mrsm sing melody lalala

mrsm sing melody --song do-re-mi

Add a Page to the Web Dashboard Use the decorators meerschaum.plugins.dash_plugin() and meerschaum.plugins.web_page() to add new pages to the web dashboard:

from meerschaum.plugins import dash_plugin, web_page

@dash_plugin
def init_dash(dash_app):

    import dash.html as html
    import dash_bootstrap_components as dbc
    from dash import Input, Output, no_update

    ### Routes to '/dash/my-page'
    @web_page('/my-page', login_required=False)
    def my_page():
        return dbc.Container([
            html.H1("Hello, World!"),
            dbc.Button("Click me", id='my-button'),
            html.Div(id="my-output-div"),
        ])

    @dash_app.callback(
        Output('my-output-div', 'children'),
        Input('my-button', 'n_clicks'),
    )
    def my_button_click(n_clicks):
        if not n_clicks:
            return no_update
        return html.P(f'You clicked {n_clicks} times!')

Submodules

meerschaum.actions
Access functions for actions and subactions.

meerschaum.config
Read and write the Meerschaum configuration registry.

meerschaum.connectors
Build connectors to interact with databases and fetch data.

meerschaum.jobs
Start background jobs.

meerschaum.plugins
Access plugin modules and other API utilties.

meerschaum.utils
Utility functions are available in several submodules:

  1#! /usr/bin/env python
  2# -*- coding: utf-8 -*-
  3# vim:fenc=utf-8
  4
  5"""
  6Copyright 2020–2026 Bennett Meares
  7
  8Licensed under the Apache License, Version 2.0 (the "License");
  9you may not use this file except in compliance with the License.
 10You may obtain a copy of the License at
 11
 12   http://www.apache.org/licenses/LICENSE-2.0
 13
 14Unless required by applicable law or agreed to in writing, software
 15distributed under the License is distributed on an "AS IS" BASIS,
 16WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 17See the License for the specific language governing permissions and
 18limitations under the License.
 19"""
 20
 21import sys as _sys
 22
 23
 24def _cli_launch() -> bool:
 25    """
 26    Whether this process is the `mrsm` command line, which does not need the façade.
 27
 28    `import meerschaum` is EAGER, exactly as it always was (the façade is imported here, under this
 29    package's import lock, so threads which import `meerschaum.*` and read `meerschaum.Pipe` at the
 30    same time cannot deadlock). The one exception is the CLI process: `mrsm` (the console script
 31    `_meerschaum_mrsm`, which sets `sys._meerschaum_cli`) and `python -m meerschaum` (which jobs
 32    run; `sys.orig_argv` starts with `-m meerschaum`) only need the extension, and importing the
 33    façade costs about 0.12 s. There the façade is imported on first use instead (PEP 562).
 34    """
 35    if getattr(_sys, '_meerschaum_cli', False):
 36        return True
 37    argv = getattr(_sys, 'orig_argv', None) or []
 38    skip = False
 39    for index, arg in enumerate(argv[1:], start=1):
 40        if skip:
 41            skip = False
 42            continue
 43        if arg == '-m':
 44            return argv[index + 1:index + 2] == ['meerschaum']
 45        if arg == '-mmeerschaum':
 46            return True
 47        if arg in ('-X', '-W', '--check-hash-based-pycs'):
 48            skip = True
 49            continue
 50        if not arg.startswith('-'):
 51            return False
 52    return False
 53
 54
 55_LAZY = _cli_launch()
 56if not _LAZY:
 57    import atexit
 58
 59    from meerschaum.utils.typing import SuccessTuple
 60    from meerschaum.utils.packages import attempt_import
 61    from meerschaum.core.Pipe import Pipe
 62    from meerschaum.core.Pipe._sync import SyncResult
 63    from meerschaum.core.Pipe._status import get_sync_statuses
 64    from meerschaum.core.Pipeline import Pipeline, Plan, RunResult, get_pipelines, get_runs
 65    from meerschaum.core.Pipeline._metrics import get_engine_metrics, initialize_engine_metrics
 66    from meerschaum.core.Team import Team, get_teams
 67    from meerschaum.plugins import Plugin
 68    from meerschaum.utils.venv import Venv
 69    from meerschaum.jobs import Job, make_executor
 70    from meerschaum.connectors import get_connector, Connector, InstanceConnector, make_connector
 71    from meerschaum.utils import get_pipes
 72    from meerschaum.utils.formatting import pprint
 73    from meerschaum._internal.docs import index as __doc__
 74    from meerschaum.config import __version__, get_config
 75    from meerschaum._internal.entry import entry
 76    from meerschaum.__main__ import _close_pools
 77
 78    atexit.register(_close_pools)
 79
 80else:
 81    import importlib as _importlib
 82
 83    _FACADE = 'meerschaum._internal._facade'
 84    _FACADE_NAMES = (
 85        'atexit',
 86        'SuccessTuple',
 87        'attempt_import',
 88        'Pipe',
 89        'SyncResult',
 90        'get_sync_statuses',
 91        'Pipeline',
 92        'Plan',
 93        'RunResult',
 94        'get_pipelines',
 95        'get_runs',
 96        'get_engine_metrics',
 97        'initialize_engine_metrics',
 98        'Team',
 99        'get_teams',
100        'Plugin',
101        'Venv',
102        'Job',
103        'make_executor',
104        'get_connector',
105        'Connector',
106        'InstanceConnector',
107        'make_connector',
108        'get_pipes',
109        'pprint',
110        '__doc__',
111        '__version__',
112        'get_config',
113        'entry',
114        '_close_pools',
115    )
116
117
118__pdoc__ = {'gui': False, 'api': False, 'core': False, '_internal': False}
119__all__ = (
120    "get_pipes",
121    "get_pipelines",
122    "get_runs",
123    "get_engine_metrics",
124    "initialize_engine_metrics",
125    "get_sync_statuses",
126    "get_teams",
127    "get_connector",
128    "get_config",
129    "Pipe",
130    "Pipeline",
131    "Plan",
132    "RunResult",
133    "SyncResult",
134    "Team",
135    "Plugin",
136    "SuccessTuple",
137    "Venv",
138    "Job",
139    "make_executor",
140    "pprint",
141    "attempt_import",
142    "actions",
143    "config",
144    "connectors",
145    "jobs",
146    "plugins",
147    "utils",
148    "Connector",
149    "InstanceConnector",
150    "make_connector",
151    "entry",
152)
153
154
155if _LAZY:
156    def _facade_module():
157        """
158        Import the façade (once) and return its module: complete, or partial when called while the
159        façade itself is being imported (a circular import sees the names bound so far, as it did
160        when they were bound in this module's body).
161        """
162        return _importlib.import_module(_FACADE)
163
164
165    def _bind_facade(facade) -> None:
166        """Bind the façade's names in this namespace once its import has finished."""
167        spec = getattr(facade, '__spec__', None)
168        if getattr(spec, '_initializing', False):
169            return
170        namespace = globals()
171        for name in _FACADE_NAMES:
172            if name in vars(facade):
173                namespace[name] = vars(facade)[name]
174
175
176    def __getattr__(name: str):
177        """Resolve a façade name, importing the façade on first use (PEP 562)."""
178        if name.startswith('__') and name.endswith('__') and name != '__version__':
179            ### Probes (`__wrapped__`, `__file__`, ...) never import anything.
180            raise AttributeError(f"module 'meerschaum' has no attribute '{name}'")
181        submodule = _sys.modules.get(f'meerschaum.{name}')
182        if submodule is not None:
183            ### A submodule being imported right now (`import meerschaum.config._paths as _paths` reads
184            ### `meerschaum.config` before the import of it has finished): never a reason to load the façade.
185            return submodule
186        if name == '__version__':
187            ### The version needs the config package, not the façade.
188            from meerschaum.config._version import __version__
189            return __version__
190        if name not in _FACADE_NAMES and name not in __all__:
191            ### Anything else is a submodule (`from meerschaum import _core`, `mrsm.core`): import just
192            ### that one; the façade stays unloaded.
193            try:
194                return _importlib.import_module(f'meerschaum.{name}')
195            except ModuleNotFoundError as e:
196                if e.name != f'meerschaum.{name}':
197                    raise
198            raise AttributeError(f"module 'meerschaum' has no attribute '{name}'")
199        facade = _facade_module()
200        _bind_facade(facade)
201        namespace = globals()
202        if name in namespace:
203            return namespace[name]
204        if name in vars(facade) and name in _FACADE_NAMES:
205            return vars(facade)[name]
206        if name in __all__:
207            ### A submodule the façade did not import as a side effect (`mrsm.plugins`, ...).
208            return _importlib.import_module(f'meerschaum.{name}')
209        raise AttributeError(f"module 'meerschaum' has no attribute '{name}'")
210
211
212    def __dir__():
213        _bind_facade(_facade_module())
214        return sorted(set(globals()) | set(__all__))
def get_pipes( connector_keys: str | List[str] | None = None, metric_keys: str | List[str] | None = None, location_keys: str | List[str] | None = None, tags: List[str] | None = None, targets: List[str] | None = None, datetime_dtypes: List[str] | None = None, params: Dict[str, Any] | None = None, mrsm_instance: str | InstanceConnector | None = None, instance: str | InstanceConnector | None = None, as_list: bool = False, as_tags_dict: bool = False, as_targets_dict: bool = False, method: str = 'registered', workers: int | None = None, debug: bool = False, _cache_parameters: bool = True, **kw: Any) -> Dict[str, Dict[str, Dict[str | None, Pipe]]] | List[Pipe] | Dict[str, Pipe]: 
 29def get_pipes(
 30    connector_keys: Union[str, List[str], None] = None,
 31    metric_keys: Union[str, List[str], None] = None,
 32    location_keys: Union[str, List[str], None] = None,
 33    tags: Optional[List[str]] = None,
 34    targets: Optional[List[str]] = None,
 35    datetime_dtypes: Optional[List[str]] = None,
 36    params: Optional[Dict[str, Any]] = None,
 37    mrsm_instance: Union[str, InstanceConnector, None] = None,
 38    instance: Union[str, InstanceConnector, None] = None,
 39    as_list: bool = False,
 40    as_tags_dict: bool = False,
 41    as_targets_dict: bool = False,
 42    method: str = 'registered',
 43    workers: Optional[int] = None,
 44    debug: bool = False,
 45    _cache_parameters: bool = True,
 46    **kw: Any
 47) -> Union[PipesDict, List[mrsm.Pipe], Dict[str, mrsm.Pipe]]:
 48    """
 49    Return a dictionary or list of `meerschaum.Pipe` objects.
 50
 51    Parameters
 52    ----------
 53    connector_keys: Union[str, List[str], None], default None
 54        String or list of connector keys.
 55        If omitted or is `'*'`, fetch all possible keys.
 56        If a string begins with `'_'`, select keys that do NOT match the string.
 57
 58    metric_keys: Union[str, List[str], None], default None
 59        String or list of metric keys. See `connector_keys` for formatting.
 60
 61    location_keys: Union[str, List[str], None], default None
 62        String or list of location keys. See `connector_keys` for formatting.
 63
 64    tags: Optional[List[str]], default None
 65        If provided, only include pipes with these tags.
 66
 67    datetime_dtypes: Optional[List[str]], default None
 68        If provided, only include pipes with the corresponding `datetime` axis dtypes.
 69        Accepted values are `datetime`, `int`, `None` (or `null`, etc.).
 70        May be negated by `_`.
 71
 72    params: Optional[Dict[str, Any]], default None
 73        Dictionary of additional parameters to search by.
 74        Params are parsed into a SQL WHERE clause.
 75        E.g. `{'a': 1, 'b': 2}` equates to `'WHERE a = 1 AND b = 2'`
 76
 77    mrsm_instance: Union[str, InstanceConnector, None], default None
 78        Connector keys for the Meerschaum instance of the pipes.
 79        Must be a `meerschaum.connectors.sql.SQLConnector.SQLConnector` or
 80        `meerschaum.connectors.api.APIConnector.APIConnector`.
 81        
 82    as_list: bool, default False
 83        If `True`, return pipes in a list instead of a hierarchical dictionary.
 84        `False` : `{connector_keys: {metric_key: {location_key: Pipe}}}`
 85        `True`  : `[Pipe]`
 86
 87    as_tags_dict: bool, default False
 88        If `True`, return a dictionary mapping tags to pipes.
 89        Pipes with multiple tags will be repeated.
 90
 91    as_targets_dict: bool, default False
 92        If `True`, return a dictionary mapping `(schema, target)` tuples to pipes.
 93        Pipes sharing the same target across different schemata are grouped separately.
 94
 95    method: str, default 'registered'
 96        Available options: `['registered', 'explicit', 'all']`
 97        If `'registered'` (default), create pipes based on registered keys in the connector's pipes table
 98        (API or SQL connector, depends on mrsm_instance).
 99        If `'explicit'`, create pipes from provided connector_keys, metric_keys, and location_keys
100        instead of consulting the pipes table. Useful for creating non-existent pipes.
101        If `'all'`, create pipes from predefined metrics and locations. Required `connector_keys`.
102        **NOTE:** Method `'all'` is not implemented!
103
104    workers: Optional[int], default None
105        If provided (and `as_tags_dict` or `as_targets_dict` is `True`), set the number of workers
106        for the pool to fetch tags or targets.
107        Only takes effect if the instance connector supports multi-threading.
108
109    **kw: Any:
110        Keyword arguments to pass to the `meerschaum.Pipe` constructor.
111
112    Returns
113    -------
114    A dictionary of dictionaries and `meerschaum.Pipe` objects
115    in the connector, metric, location hierarchy.
116    If `as_list` is `True`, return a list of `meerschaum.Pipe` objects.
117    If `as_tags_dict` is `True`, return a dictionary mapping tags to pipes.
118    If `as_targets_dict` is `True`, return a dictionary mapping targets to pipes.
119
120    Examples
121    --------
122    ```
123    >>> ### Manual definition:
124    >>> pipes = {
125    ...     <connector_keys>: {
126    ...         <metric_key>: {
127    ...             <location_key>: Pipe(
128    ...                 <connector_keys>,
129    ...                 <metric_key>,
130    ...                 <location_key>,
131    ...             ),
132    ...         },
133    ...     },
134    ... },
135    >>> ### Accessing a single pipe:
136    >>> pipes['sql:main']['weather'][None]
137    >>> ### Return a list instead:
138    >>> get_pipes(as_list=True)
139    [Pipe('sql:main', 'weather')]
140    >>> get_pipes(as_tags_dict=True)
141    {'gvl': Pipe('sql:main', 'weather')}
142    ```
143    """
144    import json
145    from collections import defaultdict
146    from meerschaum.config import get_config
147    from meerschaum.config.static import STATIC_CONFIG
148    from meerschaum.utils.warnings import error
149    from meerschaum.utils.misc import filter_keywords, separate_negation_values
150    from meerschaum.utils.pool import get_pool
151    from meerschaum.utils.pipes import replace_pipes_syntax
152    from meerschaum.utils.debug import dprint
153    from meerschaum.utils.dtypes import value_is_null, get_current_timestamp
154    from meerschaum import Pipe
155
156    negation_prefix = STATIC_CONFIG['system']['fetch_pipes_keys']['negation_prefix']
157    if datetime_dtypes:
158        if isinstance(datetime_dtypes, str):
159            datetime_dtypes = [datetime_dtypes]
160        for _dt in datetime_dtypes:
161            _clean = str(_dt).lstrip(negation_prefix).lower()
162            if _clean not in ('datetime', 'int') and not value_is_null(_clean):
163                error(f"Invalid datetime dtype '{_dt}'.")
164
165    if connector_keys is None:
166        connector_keys = []
167    if metric_keys is None:
168        metric_keys = []
169    if location_keys is None:
170        location_keys = []
171    if params is None:
172        params = {}
173    if tags is None:
174        tags = []
175
176    if isinstance(connector_keys, str):
177        connector_keys = [connector_keys]
178    if isinstance(metric_keys, str):
179        metric_keys = [metric_keys]
180    if isinstance(location_keys, str):
181        location_keys = [location_keys]
182
183    ### Get SQL or API connector (keys come from `connector.fetch_pipes_keys()`).
184    if mrsm_instance is None:
185        mrsm_instance = instance
186    if mrsm_instance is None:
187        mrsm_instance = get_config('meerschaum', 'instance', patch=True)
188    if isinstance(mrsm_instance, str):
189        from meerschaum.connectors.parse import parse_instance_keys
190        connector = parse_instance_keys(keys=mrsm_instance, debug=debug)
191    else:
192        from meerschaum.connectors import instance_types
193        valid_connector = False
194        if hasattr(mrsm_instance, 'type'):
195            if mrsm_instance.type in instance_types:
196                valid_connector = True
197        if not valid_connector:
198            error(f"Invalid instance connector: {mrsm_instance}")
199        connector = mrsm_instance
200    if debug:
201        dprint(f"Using instance connector: {connector}")
202    if not connector:
203        error(f"Could not create connector from keys: '{mrsm_instance}'")
204
205    ### Get a list of tuples for the keys needed to build pipes.
206    result = fetch_pipes_keys(
207        method,
208        connector,
209        connector_keys = connector_keys,
210        metric_keys = metric_keys,
211        location_keys = location_keys,
212        tags = tags,
213        params = params,
214        workers = workers,
215        debug = debug
216    )
217    if result is None:
218        error("Unable to build pipes!")
219    result_items: List[Tuple] = (
220        list(result.items())
221        if isinstance(result, dict)
222        else [(None, keys_tuple) for keys_tuple in result]
223    )
224
225    ### Populate the `pipes` dictionary with Pipes based on the keys
226    ### obtained from the chosen `method`.
227    in_dtypes, ex_dtypes = separate_negation_values(datetime_dtypes or [])
228    in_targets, ex_targets = separate_negation_values(targets or [])
229    pipes: PipesDict = {}
230    targets_pipes: Dict[Tuple[Optional[str], str], List[mrsm.Pipe]] = defaultdict(lambda: [])
231    connector_schema = getattr(connector, 'schema', None)
232    connector_is_sql = getattr(connector, 'type', None) == 'sql'
233    connector_flavor = getattr(connector, 'flavor', None)
234    for pipe_id, keys_tuple in result_items:
235        ck, mk, lk = keys_tuple[0], keys_tuple[1], keys_tuple[2]
236        pipe_tags_or_parameters = keys_tuple[3] if len(keys_tuple) == 4 else None
237        pipe_parameters = (
238            pipe_tags_or_parameters
239            if isinstance(pipe_tags_or_parameters, (dict, str))
240            else None
241        )
242        if isinstance(pipe_parameters, str):
243            pipe_parameters = json.loads(pipe_parameters)
244        pipe_tags = (
245            pipe_tags_or_parameters
246            if isinstance(pipe_tags_or_parameters, list)
247            else (
248                pipe_tags_or_parameters.get('tags', [])
249                if isinstance(pipe_tags_or_parameters, dict)
250                else None
251            )
252        )
253
254        pipe = Pipe(
255            ck, mk, lk,
256            mrsm_instance = connector,
257            parameters = pipe_parameters,
258            tags = pipe_tags,
259            debug = debug,
260            **filter_keywords(Pipe, **kw)
261        )
262        pipe.__dict__['_tags'] = pipe_tags
263        if pipe_id is not None:
264            pipe._cache_value('_id', pipe_id, memory_only=True, debug=debug)
265        if pipe_parameters is not None:
266            now = get_current_timestamp('ms', as_int=True) / 1000
267            full_attributes = {
268                'connector_keys': ck,
269                'metric_key': mk,
270                'location_key': lk,
271                'parameters': pipe_parameters,
272            }
273            if pipe_id is not None:
274                full_attributes['pipe_id'] = pipe_id
275            pipe._cache_value('attributes', full_attributes, memory_only=True, debug=debug)
276            pipe._cache_value('_attributes_sync_time', now, memory_only=True, debug=debug)
277
278        if datetime_dtypes or targets:
279            parameters_str = str(pipe_parameters)
280            if pipe_parameters is None or 'MRSM{' in parameters_str or 'Pipe(' in parameters_str:
281                pipe_parameters = pipe.get_parameters(debug=debug)
282
283        keep_pipe = True
284
285        if datetime_dtypes:
286            columns_val = (pipe_parameters or {}).get('columns', {}) or {}
287            dt_col = columns_val.get('datetime', None)
288            pipe_dtypes = (
289                ((pipe_parameters or {}).get('dtypes', None) or {})
290                if dt_col
291                else None
292            )
293            dt_typ = pipe_dtypes.get(dt_col, None) if dt_col else None
294
295            def _dtype_matches(clean_d):
296                if not dt_col:
297                    return value_is_null(clean_d)
298                return (
299                    (clean_d == 'int' and 'int' in str(dt_typ).lower())
300                    or
301                    (clean_d == 'datetime' and 'int' not in str(dt_typ).lower())
302                )
303
304            in_match = not in_dtypes or any(_dtype_matches(d) for d in in_dtypes)
305            ex_match = bool(ex_dtypes and any(_dtype_matches(d) for d in ex_dtypes))
306            keep_pipe = keep_pipe and in_match and not ex_match
307            if not keep_pipe:
308                continue
309
310        if targets:
311            pipe_target = pipe.target
312            in_target_match = not in_targets or any(t == pipe_target for t in in_targets)
313            ex_target_match = bool(ex_targets and any(t == pipe_target for t in ex_targets))
314            keep_pipe = keep_pipe and in_target_match and not ex_target_match
315            if not keep_pipe:
316                continue
317
318        if ck not in pipes:
319            pipes[ck] = {}
320
321        if mk not in pipes[ck]:
322            pipes[ck][mk] = {}
323
324
325        pipes[ck][mk][lk] = pipe
326
327        if as_targets_dict:
328            raw_params = pipe_parameters if isinstance(pipe_parameters, dict) else {}
329            schema = raw_params.get('schema') or connector_schema
330            explicit_target = (
331                raw_params.get('target')
332                or raw_params.get('target_name')
333                or raw_params.get('target_table')
334                or raw_params.get('target_table_name')
335            )
336            if explicit_target:
337                target_name = (
338                    replace_pipes_syntax(explicit_target, _pipe=pipe)
339                    if isinstance(explicit_target, str) and '{{' in explicit_target
340                    else explicit_target
341                )
342            else:
343                target_name = pipe._target_legacy()
344                if connector_is_sql and connector_flavor:
345                    from meerschaum.utils.sql import truncate_item_name
346                    target_name = truncate_item_name(target_name, connector_flavor)
347            targets_pipes[(schema, target_name)].append(pipe)
348
349    if not as_list and not as_tags_dict and not as_targets_dict:
350        return pipes
351
352    from meerschaum.utils.pipes import flatten_pipes_dict
353    pipes_list = flatten_pipes_dict(pipes)
354    if as_list:
355        return pipes_list
356
357    pool = get_pool(workers=(workers if connector.IS_THREAD_SAFE else 1))
358
359    def gather_pipe_tags(pipe: mrsm.Pipe) -> Tuple[mrsm.Pipe, List[str]]:
360        _tags = pipe.__dict__.get('_tags', None)
361        gathered_tags = _tags if _tags is not None else pipe.tags
362        return pipe, (gathered_tags or [])
363
364    if as_tags_dict:
365        tags_pipes = defaultdict(lambda: [])
366        pipes_tags = dict(pool.map(gather_pipe_tags, pipes_list))
367        for pipe, tags in pipes_tags.items():
368            for tag in (tags or []):
369                tags_pipes[tag].append(pipe)
370
371        return dict(tags_pipes)
372
373    if as_targets_dict:
374        return dict(targets_pipes)
375
376    raise NotImplementedError("No futher options for returning pipes.")

Return a dictionary or list of meerschaum.Pipe objects.

Parameters
  • connector_keys (Union[str, List[str], None], default None): String or list of connector keys. If omitted or is '*', fetch all possible keys. If a string begins with '_', select keys that do NOT match the string.
  • metric_keys (Union[str, List[str], None], default None): String or list of metric keys. See connector_keys for formatting.
  • location_keys (Union[str, List[str], None], default None): String or list of location keys. See connector_keys for formatting.
  • tags (Optional[List[str]], default None): If provided, only include pipes with these tags.
  • datetime_dtypes (Optional[List[str]], default None): If provided, only include pipes with the corresponding datetime axis dtypes. Accepted values are datetime, int, None (or null, etc.). May be negated by _.
  • params (Optional[Dict[str, Any]], default None): Dictionary of additional parameters to search by. Params are parsed into a SQL WHERE clause. E.g. {'a': 1, 'b': 2} equates to 'WHERE a = 1 AND b = 2'
  • mrsm_instance (Union[str, InstanceConnector, None], default None): Connector keys for the Meerschaum instance of the pipes. Must be a meerschaum.connectors.sql.SQLConnector.SQLConnector or meerschaum.connectors.api.APIConnector.APIConnector.
  • as_list (bool, default False): If True, return pipes in a list instead of a hierarchical dictionary. False : {connector_keys: {metric_key: {location_key: Pipe}}} True : [Pipe]
  • as_tags_dict (bool, default False): If True, return a dictionary mapping tags to pipes. Pipes with multiple tags will be repeated.
  • as_targets_dict (bool, default False): If True, return a dictionary mapping (schema, target) tuples to pipes. Pipes sharing the same target across different schemata are grouped separately.
  • method (str, default 'registered'): Available options: ['registered', 'explicit', 'all'] If 'registered' (default), create pipes based on registered keys in the connector's pipes table (API or SQL connector, depends on mrsm_instance). If 'explicit', create pipes from provided connector_keys, metric_keys, and location_keys instead of consulting the pipes table. Useful for creating non-existent pipes. If 'all', create pipes from predefined metrics and locations. Required connector_keys. NOTE: Method 'all' is not implemented!
  • workers (Optional[int], default None): If provided (and as_tags_dict or as_targets_dict is True), set the number of workers for the pool to fetch tags or targets. Only takes effect if the instance connector supports multi-threading.
  • **kw (Any:): Keyword arguments to pass to the meerschaum.Pipe constructor.
Returns
  • A dictionary of dictionaries and meerschaum.Pipe objects
  • in the connector, metric, location hierarchy.
  • If as_list is True, return a list of meerschaum.Pipe objects.
  • If as_tags_dict is True, return a dictionary mapping tags to pipes.
  • If as_targets_dict is True, return a dictionary mapping targets to pipes.
Examples
>>> ### Manual definition:
>>> pipes = {
...     <connector_keys>: {
...         <metric_key>: {
...             <location_key>: Pipe(
...                 <connector_keys>,
...                 <metric_key>,
...                 <location_key>,
...             ),
...         },
...     },
... },
>>> ### Accessing a single pipe:
>>> pipes['sql:main']['weather'][None]
>>> ### Return a list instead:
>>> get_pipes(as_list=True)
[Pipe('sql:main', 'weather')]
>>> get_pipes(as_tags_dict=True)
{'gvl': Pipe('sql:main', 'weather')}
def get_pipelines( instance: str | Any | None = None) -> List[Pipeline]: 
1549def get_pipelines(instance: Union[str, Any, None] = None) -> List[Pipeline]:
1550    """
1551    Return the Pipelines saved on `instance` (default: the configured instance), sorted by name.
1552
1553    Parameters
1554    ----------
1555    instance: Union[str, InstanceConnector, None], default None
1556        The instance whose saved Pipelines to list.
1557
1558    Returns
1559    -------
1560    A list of `Pipeline` objects, each with its `name` and `definition`.
1561
1562    Examples
1563    --------
1564    >>> for pipeline in mrsm.get_pipelines(instance='sql:main'):
1565    ...     print(pipeline.name, len(pipeline.members))
1566    """
1567    from meerschaum.core.Pipeline._saved import list_pipelines
1568    registry = _registry_of(instance)
1569    return [
1570        Pipeline._from_definitions([definition], registry)
1571        for _, definition in sorted(list_pipelines(registry).items())
1572    ]

Return the Pipelines saved on instance (default: the configured instance), sorted by name.

Parameters
  • instance (Union[str, InstanceConnector, None], default None): The instance whose saved Pipelines to list.
Returns
  • A list of Pipeline objects, each with its name and definition.
Examples
>>> for pipeline in mrsm.get_pipelines(instance='sql:main'):
...     print(pipeline.name, len(pipeline.members))
def get_runs( instance: str | Any | None = None, limit: int = 20, status: str | None = None, pipeline: str | Pipeline | None = None, pipe: Pipe | str | None = None, since: datetime.datetime | datetime.timedelta | str | None = None) -> List[Dict[str, Any]]: 
1575def get_runs(
1576    instance: Union[str, Any, None] = None,
1577    limit: int = 20,
1578    status: Optional[str] = None,
1579    pipeline: Union[str, Pipeline, None] = None,
1580    pipe: Union[mrsm.Pipe, str, None] = None,
1581    since: Union[datetime, timedelta, str, None] = None,
1582) -> List[Dict[str, Any]]:
1583    """
1584    Recent engine runs on `instance` (default: the configured instance), newest first.
1585
1586    Parameters
1587    ----------
1588    instance: Union[str, InstanceConnector, None], default None
1589        The instance that keeps the run history.
1590
1591    limit: int, default 20
1592        The most runs to return.
1593
1594    status: Optional[str], default None
1595        Only `'running'`, `'succeeded'` or `'failed'` runs.
1596
1597    pipeline: Union[str, Pipeline, None], default None
1598        Only the runs of this saved Pipeline (a name or a `Pipeline`).
1599
1600    pipe: Union[mrsm.Pipe, str, None], default None
1601        Only the runs which targeted or contained this pipe (a `Pipe` or a `pipe_ref`).
1602
1603    since: Union[datetime, timedelta, str, None], default None
1604        Only runs started at or after this time: a `datetime` (naive means UTC), a
1605        `timedelta` (that long ago, e.g. `timedelta(hours=1)`) or an ISO 8601 string.
1606
1607    Returns
1608    -------
1609    Run summaries: `run` (the stored row), `status`, `duration_seconds` and `counts`
1610    (nodes by status). See `Pipeline.get_runs()`.
1611
1612    Examples
1613    --------
1614    >>> from datetime import timedelta
1615    >>> failed = mrsm.get_runs(status='failed', since=timedelta(days=1))
1616    """
1617    connector = (
1618        parse_instance_keys(instance)
1619        if isinstance(instance, (str, type(None)))
1620        else instance
1621    )
1622    request = build_request(connector)
1623    request['limit'] = limit
1624    if status is not None:
1625        request['status'] = status
1626    if pipeline is not None:
1627        request['run_name'] = pipeline.name if isinstance(pipeline, Pipeline) else str(pipeline)
1628    if pipe is not None:
1629        request['pipe'] = pipe if isinstance(pipe, str) else _pipe_ref(pipe)
1630    since = since_text(since)
1631    if since is not None:
1632        request['since'] = since
1633    return call('engine_runs', request)

Recent engine runs on instance (default: the configured instance), newest first.

Parameters
  • instance (Union[str, InstanceConnector, None], default None): The instance that keeps the run history.
  • limit (int, default 20): The most runs to return.
  • status (Optional[str], default None): Only 'running', 'succeeded' or 'failed' runs.
  • pipeline (Union[str, Pipeline, None], default None): Only the runs of this saved Pipeline (a name or a Pipeline).
  • pipe (Union[mrsm.Pipe, str, None], default None): Only the runs which targeted or contained this pipe (a Pipe or a pipe_ref).
  • since (Union[datetime, timedelta, str, None], default None): Only runs started at or after this time: a datetime (naive means UTC), a timedelta (that long ago, e.g. timedelta(hours=1)) or an ISO 8601 string.
Returns

  • Run summaries (run (the stored row), status, duration_seconds and counts):

  • (nodes by status). See Pipeline.get_runs().

Examples
>>> from datetime import timedelta
>>> failed = mrsm.get_runs(status='failed', since=timedelta(days=1))
def get_engine_metrics( instance=None, begin=None, end=None, pipe=None, pipe_ref=None, pipeline=None, limit=1000, buckets=48, visible_refs=None, visible_pipelines=None, summary_only=False): 
34def get_engine_metrics(instance=None, begin=None, end=None, pipe=None, pipe_ref=None,
35                       pipeline=None, limit=1000, buckets=48, visible_refs=None,
36                       visible_pipelines=None, summary_only=False):
37    """Read numeric sync and Pipeline history on an instance.
38
39    The default window is the last 24 hours; retained history spans 30 days.
40    SQL totals and chart buckets cover the authorized window independently of the
41    detailed history limit (at most 5,000 records). ``summary_complete`` identifies
42    whole-window aggregates; ``history_truncated`` identifies capped recent history.
43    Other backends may return explicitly incomplete sample aggregates. Durations
44    are seconds, timestamps are UTC, and ``success_rate`` is a fraction from 0 to 1.
45    ``rows`` counts changed rows (inserted + updated + upserted + deleted), so
46    replacement syncs count both deletions and new rows; unchanged syncs count zero.
47    Local callers are trusted. Hosts must supply fresh ``visible_refs`` and
48    ``visible_pipelines`` before aggregation; empty lists see nothing. API instances
49    additionally enforce their current principal's scopes, teams and object access.
50    """
51    from meerschaum import _core
52    instance = _instance(instance)
53    carrier, context = _context(instance)
54    if pipe is not None and pipe_ref is not None:
55        raise ValueError('Supply pipe or pipe_ref, not both.')
56    reference = pipe_ref if pipe_ref is not None else pipe
57    if reference is not None and not isinstance(reference, str):
58        reference = '|'.join((str(reference.instance_keys), str(reference.connector_keys),
59                              str(reference.metric_key), str(reference.location_key)))
60    request = {
61        'instance': str(instance), 'begin': _time(begin), 'end': _time(end),
62        'pipe_ref': reference, 'pipeline': getattr(pipeline, 'name', pipeline),
63        'limit': limit, 'buckets': buckets,
64        'summary_only': bool(summary_only),
65        'visible_refs': list(visible_refs) if visible_refs is not None else None,
66        'visible_pipelines': list(visible_pipelines) if visible_pipelines is not None else None,
67    }
68    return json.loads(_core.engine_metrics(json.dumps(request), carrier, context))

Read numeric sync and Pipeline history on an instance.

The default window is the last 24 hours; retained history spans 30 days. SQL totals and chart buckets cover the authorized window independently of the detailed history limit (at most 5,000 records). summary_complete identifies whole-window aggregates; history_truncated identifies capped recent history. Other backends may return explicitly incomplete sample aggregates. Durations are seconds, timestamps are UTC, and success_rate is a fraction from 0 to 1. rows counts changed rows (inserted + updated + upserted + deleted), so replacement syncs count both deletions and new rows; unchanged syncs count zero. Local callers are trusted. Hosts must supply fresh visible_refs and visible_pipelines before aggregation; empty lists see nothing. API instances additionally enforce their current principal's scopes, teams and object access.

def initialize_engine_metrics(instance=None): 
71def initialize_engine_metrics(instance=None):
72    """Create empty native metrics/run/node storage and indexes for dashboards.
73
74    API servers initialize their served instance during startup. A remote initialization
75    requires a full administrator; local access uses the instance's own credentials.
76    """
77    from meerschaum import _core
78    instance = _instance(instance)
79    carrier, context = _context(instance)
80    _core.initialize_engine_metrics(json.dumps({'instance': str(instance)}), carrier, context)

Create empty native metrics/run/node storage and indexes for dashboards.

API servers initialize their served instance during startup. A remote initialization requires a full administrator; local access uses the instance's own credentials.

def get_sync_statuses( instance: str | InstanceConnector | None = None, debug: bool = False) -> Dict[str, Dict[str, Any]]: 
 71def get_sync_statuses(
 72    instance: Union[str, InstanceConnector, None] = None,
 73    debug: bool = False,
 74) -> Dict[str, Dict[str, Any]]:
 75    """
 76    Return the latest recorded sync of every pipe on an instance.
 77
 78    Parameters
 79    ----------
 80    instance: Union[str, InstanceConnector, None], default None
 81        The instance connector or its keys. Defaults to the configured instance
 82        (`meerschaum:instance`).
 83
 84    debug: bool, default False
 85        Verbosity toggle.
 86
 87    Returns
 88    -------
 89    A dictionary of `pipe_ref` (`'instance|connector|metric|location'`) to the pipe's status
 90    dictionary (see `meerschaum.core.Pipe._status`). Pipes without a recorded sync are absent.
 91    Reading from an unreachable instance raises an exception.
 92
 93    Examples
 94    --------
 95    >>> statuses = mrsm.get_sync_statuses('sql:local')
 96    >>> failing = [s for s in statuses.values() if s['consecutive_failures'] > 0]
 97    """
 98    from meerschaum import _core
 99    from meerschaum.config import get_config
100    if instance is None:
101        instance = get_config('meerschaum', 'instance')
102    ### A pipe only carries the instance to the binding: it is never registered or read.
103    carrier = mrsm.Pipe('mrsm', 'engine_sync_status', instance=instance, temporary=True)
104    statuses = _core.get_sync_statuses(
105        carrier,
106        _status_context(carrier, debug=debug),
107        debug=debug,
108    )
109    return {status['pipe_ref']: status for status in statuses}

Return the latest recorded sync of every pipe on an instance.

Parameters
  • instance (Union[str, InstanceConnector, None], default None): The instance connector or its keys. Defaults to the configured instance (meerschaum:instance).
  • debug (bool, default False): Verbosity toggle.
Returns
  • A dictionary of pipe_ref ('instance|connector|metric|location') to the pipe's status
  • dictionary (see meerschaum.core.Pipe._status). Pipes without a recorded sync are absent.
  • Reading from an unreachable instance raises an exception.
Examples
>>> statuses = mrsm.get_sync_statuses('sql:local')
>>> failing = [s for s in statuses.values() if s['consecutive_failures'] > 0]
def get_teams( instance: str | InstanceConnector | None = None, debug: bool = False) -> List[Team]: 
367def get_teams(
368    instance: Union[str, 'mrsm.connectors.InstanceConnector', None] = None,
369    debug: bool = False,
370) -> List[Team]:
371    """
372    Return the teams of an instance, `Everyone` first.
373
374    Parameters
375    ----------
376    instance: Union[str, InstanceConnector, None], default None
377        The instance (the configured instance by default).
378
379    Returns
380    -------
381    A list of `Team` objects.
382
383    Examples
384    --------
385    >>> [team.name for team in mrsm.get_teams('sql:local')]
386    ['Everyone', 'ops']
387    """
388    if instance is None:
389        from meerschaum.config import get_config
390        instance = get_config('meerschaum', 'instance', patch=True)
391    if isinstance(instance, str):
392        from meerschaum.connectors.parse import parse_instance_keys
393        instance = parse_instance_keys(instance)
394    teams = []
395    for record in (instance.get_teams(debug=debug) or []):
396        if isinstance(record, Team):
397            teams.append(record)
398            continue
399        if not isinstance(record, dict):
400            record = {'name': str(getattr(record, 'name', record))}
401        team = Team(str(record.get('name', '')), instance=instance)
402        team._apply_record(record)
403        team._loaded = True
404        teams.append(team)
405    return teams

Return the teams of an instance, Everyone first.

Parameters
  • instance (Union[str, InstanceConnector, None], default None): The instance (the configured instance by default).
Returns
  • A list of Team objects.
Examples
>>> [team.name for team in mrsm.get_teams('sql:local')]
['Everyone', 'ops']
def get_connector( type: str = None, label: str = None, refresh: bool = False, debug: bool = False, _load_plugins: bool = True, _warn_undefined: bool = True, **kw: Any) -> Connector: 
 76def get_connector(
 77    type: str = None,
 78    label: str = None,
 79    refresh: bool = False,
 80    debug: bool = False,
 81    _load_plugins: bool = True,
 82    _warn_undefined: bool = True,
 83    **kw: Any
 84) -> Connector:
 85    """
 86    Return existing connector or create new connection and store for reuse.
 87    
 88    You can create new connectors if enough parameters are provided for the given type and flavor.
 89
 90    Parameters
 91    ----------
 92    type: Optional[str], default None
 93        Connector type (sql, api, etc.).
 94        Defaults to the type of the configured `instance_connector`.
 95
 96    label: Optional[str], default None
 97        Connector label (e.g. main). Defaults to `'main'`.
 98
 99    refresh: bool, default False
100        Refresh the Connector instance / construct new object. Defaults to `False`.
101
102    kw: Any
103        Other arguments to pass to the Connector constructor.
104        If the Connector has already been constructed and new arguments are provided,
105        `refresh` is set to `True` and the old Connector is replaced.
106
107    Returns
108    -------
109    A new Meerschaum connector (e.g. `meerschaum.connectors.api.APIConnector`,
110    `meerschaum.connectors.sql.SQLConnector`).
111    
112    Examples
113    --------
114    The following parameters would create a new
115    `meerschaum.connectors.sql.SQLConnector` that isn't in the configuration file.
116
117    ```
118    >>> conn = get_connector(
119    ...     type = 'sql',
120    ...     label = 'newlabel',
121    ...     flavor = 'sqlite',
122    ...     database = '/file/path/to/database.db'
123    ... )
124    >>>
125    ```
126
127    """
128    from meerschaum.connectors.parse import parse_instance_keys
129    from meerschaum.config import get_config
130    from meerschaum._internal.static import STATIC_CONFIG
131    from meerschaum.utils.warnings import warn
132    global _loaded_plugin_connectors
133    if isinstance(type, str) and not label and ':' in type:
134        type, label = type.split(':', maxsplit=1)
135
136    if _load_plugins:
137        with _locks['_loaded_plugin_connectors']:
138            if not _loaded_plugin_connectors:
139                load_plugin_connectors()
140                _load_builtin_custom_connectors()
141                _loaded_plugin_connectors = True
142
143    if type is None and label is None:
144        default_instance_keys = get_config('meerschaum', 'instance', patch=True)
145        ### recursive call to get_connector
146        return parse_instance_keys(default_instance_keys)
147
148    ### NOTE: the default instance connector may not be main.
149    ### Only fall back to 'main' if the type is provided by the label is omitted.
150    label = label if label is not None else STATIC_CONFIG['connectors']['default_label']
151
152    ### A type declared by a native (Rust) plugin is registered on first use.
153    if type not in connectors and _load_plugins and isinstance(type, str):
154        _load_native_connector_type(type)
155
156    ### type might actually be a label. Check if so and raise a warning.
157    if type not in connectors:
158        possibilities, poss_msg = [], ""
159        for _type in get_config('meerschaum', 'connectors'):
160            if type in get_config('meerschaum', 'connectors', _type):
161                possibilities.append(f"{_type}:{type}")
162        if len(possibilities) > 0:
163            poss_msg = " Did you mean"
164            for poss in possibilities[:-1]:
165                poss_msg += f" '{poss}',"
166            if poss_msg.endswith(','):
167                poss_msg = poss_msg[:-1]
168            if len(possibilities) > 1:
169                poss_msg += " or"
170            poss_msg += f" '{possibilities[-1]}'?"
171
172        warn(f"Cannot create Connector of type '{type}'." + poss_msg, stack=False)
173        return None
174
175    if 'sql' not in types:
176        from meerschaum.connectors.plugin import PluginConnector
177        from meerschaum.connectors.valkey import ValkeyConnector
178        with _locks['types']:
179            types.update({
180                'api': APIConnector,
181                'sql': SQLConnector,
182                'plugin': PluginConnector,
183                'valkey': ValkeyConnector,
184            })
185
186    ### determine if we need to call the constructor
187    if not refresh:
188        ### see if any user-supplied arguments differ from the existing instance
189        if label in connectors[type]:
190            warning_message = None
191            for attribute, value in kw.items():
192                if attribute not in connectors[type][label].meta:
193                    import inspect
194                    cls = connectors[type][label].__class__
195                    cls_init_signature = inspect.signature(cls)
196                    cls_init_params = cls_init_signature.parameters
197                    if attribute not in cls_init_params:
198                        warning_message = (
199                            f"Received new attribute '{attribute}' not present in connector " +
200                            f"{connectors[type][label]}.\n"
201                        )
202                elif connectors[type][label].__dict__[attribute] != value:
203                    warning_message = (
204                        f"Mismatched values for attribute '{attribute}' in connector "
205                        + f"'{connectors[type][label]}'.\n" +
206                        f"  - Keyword value: '{value}'\n" +
207                        f"  - Existing value: '{connectors[type][label].__dict__[attribute]}'\n"
208                    )
209            if warning_message is not None:
210                warning_message += (
211                    "\nSetting `refresh` to True and recreating connector with type:"
212                    + f" '{type}' and label '{label}'."
213                )
214                refresh = True
215                warn(warning_message)
216        else: ### connector doesn't yet exist
217            refresh = True
218
219    ### only create an object if refresh is True
220    ### (can be manually specified, otherwise determined above)
221    if refresh:
222        ### A connector defined nowhere: say where connectors come from (not "Missing flavor").
223        undefined = _undefined_connector_message(type, label, kw)
224        if undefined is not None:
225            if _warn_undefined:
226                warn(undefined, stack=False)
227            return None
228        with _locks['connectors']:
229            try:
230                ### will raise an error if configuration is incorrect / missing
231                conn = types[type](label=label, **kw)
232                connectors[type][label] = conn
233            except InvalidAttributesError as ie:
234                warn(
235                    f"Incorrect attributes for connector '{type}:{label}'.\n"
236                    + str(ie),
237                    stack = False,
238                )
239                conn = None
240            except Exception as e:
241                from meerschaum.utils.formatting import print_exception
242                print_exception()
243                warn(
244                    f"Exception when creating connector '{type}:{label}'.\n" + str(e),
245                    stack = False,
246                )
247                conn = None
248        if conn is None:
249            return None
250
251    return connectors[type][label]

Return existing connector or create new connection and store for reuse.

You can create new connectors if enough parameters are provided for the given type and flavor.

Parameters
  • type (Optional[str], default None): Connector type (sql, api, etc.). Defaults to the type of the configured instance_connector.
  • label (Optional[str], default None): Connector label (e.g. main). Defaults to 'main'.
  • refresh (bool, default False): Refresh the Connector instance / construct new object. Defaults to False.
  • kw (Any): Other arguments to pass to the Connector constructor. If the Connector has already been constructed and new arguments are provided, refresh is set to True and the old Connector is replaced.
Returns
Examples

The following parameters would create a new meerschaum.connectors.sql.SQLConnector that isn't in the configuration file.

>>> conn = get_connector(
...     type = 'sql',
...     label = 'newlabel',
...     flavor = 'sqlite',
...     database = '/file/path/to/database.db'
... )
>>>
def get_config( *keys: str, patch: bool = True, substitute: bool = True, sync_files: bool = True, write_missing: bool = True, as_tuple: bool = False, warn: bool = True, debug: bool = False) -> Any: 
116def get_config(
117    *keys: str,
118    patch: bool = True,
119    substitute: bool = True,
120    sync_files: bool = True,
121    write_missing: bool = True,
122    as_tuple: bool = False,
123    warn: bool = True,
124    debug: bool = False
125) -> Any:
126    """
127    Return the Meerschaum configuration dictionary.
128    If positional arguments are provided, index by the keys.
129    Raises a warning if invalid keys are provided.
130
131    Parameters
132    ----------
133    keys: str:
134        List of strings to index.
135
136    patch: bool, default True
137        If `True`, patch missing default keys into the config directory.
138        Defaults to `True`.
139
140    sync_files: bool, default True
141        If `True`, sync files if needed.
142        Defaults to `True`.
143
144    write_missing: bool, default True
145        If `True`, write default values when the main config files are missing.
146        Defaults to `True`.
147
148    substitute: bool, default True
149        If `True`, subsitute 'MRSM{}' values.
150        Defaults to `True`.
151
152    as_tuple: bool, default False
153        If `True`, return a tuple of type (success, value).
154        Defaults to `False`.
155        
156    Returns
157    -------
158    The value in the configuration directory, indexed by the provided keys.
159
160    Examples
161    --------
162    >>> get_config('meerschaum', 'instance')
163    'sql:main'
164    >>> get_config('does', 'not', 'exist')
165    UserWarning: Invalid keys in config: ('does', 'not', 'exist')
166    """
167    return _core.get_config(
168        *keys, patch=patch, substitute=substitute, sync_files=sync_files,
169        write_missing=write_missing, as_tuple=as_tuple, warn=warn, debug=debug,
170    )

Return the Meerschaum configuration dictionary. If positional arguments are provided, index by the keys. Raises a warning if invalid keys are provided.

Parameters
  • keys (str:): List of strings to index.
  • patch (bool, default True): If True, patch missing default keys into the config directory. Defaults to True.
  • sync_files (bool, default True): If True, sync files if needed. Defaults to True.
  • write_missing (bool, default True): If True, write default values when the main config files are missing. Defaults to True.
  • substitute (bool, default True): If True, subsitute 'MRSM{}' values. Defaults to True.
  • as_tuple (bool, default False): If True, return a tuple of type (success, value). Defaults to False.
Returns
  • The value in the configuration directory, indexed by the provided keys.
Examples
>>> get_config('meerschaum', 'instance')
'sql:main'
>>> get_config('does', 'not', 'exist')
UserWarning: Invalid keys in config: ('does', 'not', 'exist')
class Pipe: 
 65class Pipe:
 66    """
 67    Access Meerschaum pipes via Pipe objects.
 68    
 69    Pipes are identified by the following:
 70
 71    1. Connector keys (e.g. `'sql:main'`)
 72    2. Metric key (e.g. `'weather'`)
 73    3. Location (optional; e.g. `None`)
 74    
 75    A pipe's connector keys correspond to a data source, and when the pipe is synced,
 76    its `fetch` definition is evaluated and executed to produce new data.
 77    
 78    Alternatively, new data may be directly synced via `pipe.sync()`:
 79    
 80    ```
 81    >>> from meerschaum import Pipe
 82    >>> pipe = Pipe('csv', 'weather')
 83    >>>
 84    >>> import pandas as pd
 85    >>> df = pd.read_csv('weather.csv')
 86    >>> pipe.sync(df)
 87    ```
 88    """
 89
 90    from ._fetch import (
 91        fetch,
 92        get_backtrack_interval,
 93    )
 94    from ._data import (
 95        get_data,
 96        get_backtrack_data,
 97        get_rowcount,
 98        get_size,
 99        get_doc,
100        get_docs,
101        get_value,
102        get_chunk_interval,
103        get_chunk_bounds,
104        get_chunk_bounds_batches,
105        parse_date_bounds,
106    )
107    from ._register import register
108    from ._attributes import (
109        attributes,
110        parameters,
111        columns,
112        indices,
113        indexes,
114        dtypes,
115        autoincrement,
116        autotime,
117        upsert,
118        static,
119        tzinfo,
120        enforce,
121        null_indices,
122        mixed_numerics,
123        get_columns,
124        get_columns_types,
125        get_columns_indices,
126        get_indices,
127        get_parameters,
128        get_dtypes,
129        update_parameters,
130        validate_indices,
131        tags,
132        get_id,
133        id,
134        get_val_column,
135        parents,
136        parent,
137        children,
138        child,
139        reference,
140        references,
141        target,
142        _target_legacy,
143        guess_datetime,
144        precision,
145        get_precision,
146    )
147    from ._cache import (
148        _get_cache_connector,
149        _cache_value,
150        _get_cached_value,
151        _invalidate_cache,
152        _get_cache_dir_path,
153        _write_cache_key,
154        _write_cache_file,
155        _write_cache_conn_key,
156        _read_cache_key,
157        _read_cache_file,
158        _read_cache_conn_key,
159        _load_cache_keys,
160        _load_cache_files,
161        _load_cache_conn_keys,
162        _get_cache_keys,
163        _get_cache_file_keys,
164        _get_cache_conn_keys,
165        _clear_cache_key,
166        _clear_cache_file,
167        _clear_cache_conn_key,
168    )
169    from ._show import show
170    from ._edit import edit, edit_definition, update
171    from ._sync import (
172        sync,
173        get_sync_time,
174        exists,
175        filter_existing,
176        _get_chunk_label,
177        get_num_workers,
178    )
179    from ._verify import (
180        verify,
181        verify_schema,
182        get_bound_interval,
183        get_bound_time,
184    )
185    from ._delete import delete
186    from ._drop import drop, drop_indices
187    from ._compress import compress, decompress
188    from ._maintenance import vacuum, analyze, repartition
189    from ._index import create_indices
190    from ._clear import clear
191    from ._deduplicate import deduplicate
192    from ._bootstrap import bootstrap
193    from ._dtypes import enforce_dtypes, infer_dtypes
194    from ._copy import copy_to
195    from ._status import get_sync_status
196    from ._access import (
197        access,
198        owner,
199        team,
200        visibility,
201        edit_mode,
202        locks,
203        lock,
204        unlock,
205        set_access,
206        _lock_refusal,
207        _edit_lock_refusal,
208    )
209
210    def __init__(
211        self,
212        connector: str = '',
213        metric: str = '',
214        location: Optional[str] = None,
215        parameters: Optional[Dict[str, Any]] = None,
216        columns: Union[Dict[str, str], List[str], None] = None,
217        indices: Optional[Dict[str, Union[str, List[str]]]] = None,
218        tags: Optional[List[str]] = None,
219        target: Optional[str] = None,
220        dtypes: Optional[Dict[str, str]] = None,
221        instance: Optional[Union[str, InstanceConnector]] = None,
222        upsert: Optional[bool] = None,
223        autoincrement: Optional[bool] = None,
224        autotime: Optional[bool] = None,
225        precision: Union[str, Dict[str, Union[str, int]], None] = None,
226        static: Optional[bool] = None,
227        enforce: Optional[bool] = None,
228        null_indices: Optional[bool] = None,
229        mixed_numerics: Optional[bool] = None,
230        compress: Union[bool, Dict[str, Any], None] = None,
231        temporary: bool = False,
232        cache: Optional[bool] = None,
233        cache_connector_keys: Optional[str] = None,
234        references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
235        reference: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
236        parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
237        parent: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
238        children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
239        child: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
240        mrsm_instance: Optional[Union[str, InstanceConnector]] = None,
241        connector_keys: Optional[str] = None,
242        metric_key: Optional[str] = None,
243        location_key: Optional[str] = None,
244        instance_keys: Optional[str] = None,
245        indexes: Union[Dict[str, str], List[str], None] = None,
246        hypercore: Optional[bool] = None,
247        debug: bool = False,
248    ):
249        """
250        Parameters
251        ----------
252        connector: str
253            Keys for the pipe's source connector, e.g. `'sql:main'`.
254
255        metric: str
256            Label for the pipe's contents, e.g. `'weather'`.
257
258        location: str, default None
259            Label for the pipe's location. Defaults to `None`.
260
261        parameters: Optional[Dict[str, Any]], default None
262            Optionally set a pipe's parameters from the constructor,
263            e.g. columns and other attributes.
264            You can edit these parameters with `edit pipes`.
265
266        columns: Union[Dict[str, str], List[str], None], default None
267            Set the `columns` dictionary of `parameters`.
268            If `parameters` is also provided, this dictionary is added under the `'columns'` key.
269
270        indices: Optional[Dict[str, Union[str, List[str]]]], default None
271            Set the `indices` dictionary of `parameters`.
272            If `parameters` is also provided, this dictionary is added under the `'indices'` key.
273
274        tags: Optional[List[str]], default None
275            A list of strings to be added under the `'tags'` key of `parameters`.
276            You can select pipes with certain tags using `--tags`.
277
278        dtypes: Optional[Dict[str, str]], default None
279            Set the `dtypes` dictionary of `parameters`.
280            If `parameters` is also provided, this dictionary is added under the `'dtypes'` key.
281
282        mrsm_instance: Optional[Union[str, InstanceConnector]], default None
283            Connector for the Meerschaum instance where the pipe resides.
284            Defaults to the preconfigured default instance (`'sql:main'`).
285
286        instance: Optional[Union[str, InstanceConnector]], default None
287            Alias for `mrsm_instance`. If `mrsm_instance` is supplied, this value is ignored.
288
289        upsert: Optional[bool], default None
290            If `True`, set `upsert` to `True` in the parameters.
291
292        autoincrement: Optional[bool], default None
293            If `True`, set `autoincrement` in the parameters.
294
295        autotime: Optional[bool], default None
296            If `True`, set `autotime` in the parameters.
297
298        precision: Union[str, Dict[str, Union[str, int]], None], default None
299            If provided, set `precision` in the parameters.
300            This may be either a string (the precision unit) or a dictionary of in the form
301            `{'unit': <unit>, 'interval': <interval>}`.
302            Default is determined by the `datetime` column dtype
303            (e.g. `datetime64[us]` is `microsecond` precision).
304
305        static: Optional[bool], default None
306            If `True`, set `static` in the parameters.
307
308        enforce: Optional[bool], default None
309            If `False`, skip data type enforcement.
310            Default behavior is `True`.
311
312        null_indices: Optional[bool], default None
313            Set to `False` if there will be no null values in the index columns.
314            Defaults to `True`.
315
316        mixed_numerics: bool, default None
317            If `True`, integer columns will be converted to `numeric` when floats are synced.
318            Set to `False` to disable this behavior.
319            Defaults to `True`.
320
321        compress: Union[bool, Dict[str, Any], None], default None
322            If `True` (or a dictionary of compression settings), mark the pipe for compression.
323            For TimescaleDB hypertables, a columnstore (compression) policy is installed
324            automatically on sync. A dictionary may override `segmentby`, `orderby`, and `after`.
325            Defaults to `False`.
326
327        hypercore: Optional[bool], default None
328            Set `hypercore` in the parameters. For TimescaleDB hypertables, the Hypercore
329            columnstore is enabled at table creation by default (declaring
330            `segmentby`/`orderby` in `CREATE TABLE`), which causes TimescaleDB to auto-create a
331            columnstore policy. Set to `False` for a plain row-store hypertable.
332            Has no effect unless the pipe is a hypertable (`hypertable`, default `True`).
333
334        temporary: bool, default False
335            If `True`, prevent instance tables (pipes, users, plugins) from being created.
336
337        cache: Optional[bool], default None
338            If `True`, cache the pipe's metadata to disk (in addition to in-memory caching).
339            If `cache` is not explicitly `True`, it is set to `False` if `temporary` is `True`.
340            Defaults to `True` (from `None`).
341
342        cache_connector_keys: Optional[str], default None
343            If provided, use the keys to a Valkey connector (e.g. `valkey:main`).
344
345        references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
346            If provided, inherit the parameters of the reference Pipe(s).
347            May be equal to a string of the Pipe constructor, a dictionary of constructor keys,
348            a Pipe itself, or a list of any of these values.
349
350        parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
351            Set references for parent pipes. See `references` for values.
352
353        children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
354            Set references for child pipes. See `references` for values.
355
356        """
357        from meerschaum.utils.warnings import error, warn
358        if (not connector and not connector_keys) or (not metric and not metric_key):
359            error(
360                "Please provide strings for the connector and metric\n    "
361                + "(first two positional arguments)."
362            )
363
364        ### Fall back to legacy `location_key` just in case.
365        if not location:
366            location = location_key
367
368        if not connector:
369            connector = connector_keys
370
371        if not metric:
372            metric = metric_key
373
374        if location in ('[None]', 'None'):
375            location = None
376
377        from meerschaum._internal.static import STATIC_CONFIG
378        negation_prefix = STATIC_CONFIG['system']['fetch_pipes_keys']['negation_prefix']
379        for k in (connector, metric, location, *(tags or [])):
380            if str(k).startswith(negation_prefix):
381                error(f"A pipe's keys and tags cannot start with the prefix '{negation_prefix}'.")
382
383        self._connector_keys = str(connector)
384        self._connector_key = self._connector_keys ### Alias
385        self._metric_key = metric
386        self._location_key = location
387        self.temporary = temporary
388        self.cache = (
389            cache
390            if cache is not None
391            else ((not temporary) and get_config('pipes', 'cache', 'enabled', warn=False))
392        )
393        self.cache_connector_keys = (
394            str(cache_connector_keys)
395            if cache_connector_keys is not None
396            else None
397        )
398        self.debug = debug
399
400        _mrsm_instance = mrsm_instance if mrsm_instance is not None else (instance or instance_keys)
401        if _mrsm_instance is None:
402            _mrsm_instance = get_config('meerschaum', 'instance', patch=True)
403
404        if not isinstance(_mrsm_instance, str):
405            self._instance_connector = _mrsm_instance
406            self._instance_keys = str(_mrsm_instance)
407        else:
408            self._instance_keys = _mrsm_instance
409
410        if self._instance_keys == 'sql:memory':
411            self.cache = False
412
413        from meerschaum import _core
414        from ._live_parameters import Attrs, new_raw_root
415        ### Keys and raw parameters live in a `_core.Pipe`; `_raw_root` is a read-through
416        ### view of its raw map. Constructing never touches the instance.
417        self._core_pipe = _core.Pipe(
418            self._connector_keys,
419            self._metric_key,
420            self._location_key,
421            self._instance_keys,
422            None,
423        )
424        self._raw_root = new_raw_root(self._core_pipe)
425        self._attributes = Attrs(
426            self,
427            {
428                'connector_keys': self._connector_keys,
429                'metric_key': self._metric_key,
430                'location_key': self._location_key,
431                'parameters': {},
432            },
433            self._raw_root,
434        )
435
436        ### only set parameters if values are provided
437        if isinstance(parameters, dict):
438            self._attributes['parameters'] = parameters
439        else:
440            if parameters is not None:
441                warn(f"The provided parameters are of invalid type '{type(parameters)}'.")
442            self._attributes['parameters'] = {}
443
444        columns = columns or self._attributes.get('parameters', {}).get('columns', None)
445        if isinstance(columns, (list, tuple)):
446            columns = {str(col): str(col) for col in columns}
447        if isinstance(columns, dict):
448            self._attributes['parameters']['columns'] = columns
449        elif isinstance(columns, str) and 'Pipe(' in columns:
450            pass
451        elif columns is not None:
452            warn(f"The provided columns are of invalid type '{type(columns)}'.")
453
454        indices = (
455            indices
456            or indexes
457            or self._attributes.get('parameters', {}).get('indices', None)
458            or self._attributes.get('parameters', {}).get('indexes', None)
459        )
460        if isinstance(indices, dict):
461            indices_key = (
462                'indexes'
463                if 'indexes' in self._attributes['parameters']
464                else 'indices'
465            )
466            self._attributes['parameters'][indices_key] = indices
467
468        if isinstance(tags, (list, tuple)):
469            self._attributes['parameters']['tags'] = tags
470        elif tags is not None:
471            warn(f"The provided tags are of invalid type '{type(tags)}'.")
472
473        if isinstance(target, str):
474            self._attributes['parameters']['target'] = target
475        elif target is not None:
476            warn(f"The provided target is of invalid type '{type(target)}'.")
477
478        if isinstance(dtypes, dict):
479            self._attributes['parameters']['dtypes'] = dtypes
480        elif dtypes is not None:
481            warn(f"The provided dtypes are of invalid type '{type(dtypes)}'.")
482
483        if isinstance(upsert, bool):
484            self._attributes['parameters']['upsert'] = upsert
485
486        if isinstance(autoincrement, bool):
487            self._attributes['parameters']['autoincrement'] = autoincrement
488
489        if isinstance(autotime, bool):
490            self._attributes['parameters']['autotime'] = autotime
491
492        if isinstance(precision, dict):
493            self._attributes['parameters']['precision'] = precision
494        elif isinstance(precision, str):
495            self._attributes['parameters']['precision'] = {'unit': precision}
496
497        if isinstance(static, bool):
498            self._attributes['parameters']['static'] = static
499            self._static = static
500
501        if isinstance(enforce, bool):
502            self._attributes['parameters']['enforce'] = enforce
503
504        if isinstance(null_indices, bool):
505            self._attributes['parameters']['null_indices'] = null_indices
506
507        if isinstance(mixed_numerics, bool):
508            self._attributes['parameters']['mixed_numerics'] = mixed_numerics
509
510        if isinstance(compress, (bool, dict)):
511            self._attributes['parameters']['compress'] = compress
512
513        if isinstance(hypercore, bool):
514            self._attributes['parameters']['hypercore'] = hypercore
515
516        self._cache_locks = collections.defaultdict(lambda: threading.RLock())
517        self._attributes_lock = threading.RLock()
518
519        if references is not None or reference is not None:
520            reference_vals = references if references is not None else reference
521            self.references = reference_vals
522
523        if parents is not None or parent is not None:
524            parent_vals = parents if parents is not None else parent
525            self.parents = parent_vals
526
527        if children is not None or child is not None:
528            children_vals = children if children is not None else child
529            self.children = children_vals
530
531    @property
532    def metric_key(self) -> str:
533        """
534        Return the pipe's metric key.
535        """
536        return self._core_pipe.metric_key
537
538    @property
539    def metric(self) -> str:
540        """
541        Return the pipe's metric key.
542        """
543        return self.metric_key
544
545    @property
546    def location_key(self) -> Union[str, None]:
547        """
548        Return the pipe's location key.
549        """
550        return self._core_pipe.location_key
551
552    @property
553    def location(self) -> Union[str, None]:
554        """
555        Return the pipe's location key.
556        """
557        return self.location_key
558
559    @property
560    def meta(self):
561        """
562        Return the four keys needed to reconstruct this pipe.
563        """
564        return {
565            'connector_keys': self.connector_keys,
566            'metric_key': self.metric_key,
567            'location_key': self.location_key,
568            'instance_keys': self.instance_keys,
569        }
570
571    def keys(self) -> List[str]:
572        """
573        Return the ordered keys for this pipe.
574        """
575        return {
576            key: val
577            for key, val in self.meta.items()
578            if key != 'instance'
579        }
580
581    @property
582    def instance_keys(self) -> str:
583        """
584        Return the pipe's instance keys.
585        """
586        return self._core_pipe.instance_keys
587
588    @property
589    def instance(self) -> Union[InstanceConnector, str]:
590        """
591        Return the pipe's instance connector or keys.
592        """
593        conn = self.instance_connector
594        if conn is None:
595            return self.instance_keys
596        return conn
597
598    @property
599    def instance_connector(self) -> Union[InstanceConnector, None]:
600        """
601        The instance connector on which this pipe resides.
602        """
603        if '_instance_connector' not in self.__dict__:
604            from meerschaum.connectors.parse import parse_instance_keys
605            conn = parse_instance_keys(self.instance_keys)
606            if conn:
607                self._instance_connector = conn
608            else:
609                return None
610        return self._instance_connector
611
612    @property
613    def connector_keys(self) -> str:
614        """
615        Return the pipe's connector keys.
616        """
617        return self._core_pipe.connector_keys
618
619    @property
620    def connector_key(self) -> str:
621        """
622        Legacy: use `Pipe.connector_keys` instead.
623        """
624        return self.connector_keys
625
626    @property
627    def connector(self) -> Union['Connector', str]:
628        """
629        The connector to the data source.
630        """
631        if '_connector' not in self.__dict__:
632            from meerschaum.connectors.parse import parse_instance_keys
633            import warnings
634            with warnings.catch_warnings():
635                warnings.simplefilter('ignore')
636                try:
637                    conn = parse_instance_keys(self.connector_keys)
638                except Exception:
639                    conn = None
640            if conn:
641                self._connector = conn
642            else:
643                return self._connector_keys
644        return self._connector
645
646    def __str__(self, ansi: bool=False):
647        if not ansi:
648            return self._core_pipe.__str__()
649        return pipe_repr(self, ansi=ansi)
650
651    def __eq__(self, other):
652        ### Duck typed: only another façade pipe backed by `_core.Pipe` compares in Rust.
653        other_core = getattr(other, '__dict__', {}).get('_core_pipe', None)
654        if other_core is not None:
655            return isinstance(self, type(other)) and self._core_pipe == other_core
656        try:
657            return (
658                isinstance(self, type(other))
659                and self.connector_keys == other.connector_keys
660                and self.metric_key == other.metric_key
661                and self.location_key == other.location_key
662                and self.instance_keys == other.instance_keys
663            )
664        except Exception:
665            return False
666
667    def __hash__(self):
668        return hash(self._core_pipe)
669
670    def __repr__(self, ansi: bool=True, **kw) -> str:
671        if not hasattr(sys, 'ps1'):
672            ansi = False
673
674        if not ansi and not kw:
675            return self._core_pipe.__str__()
676        return pipe_repr(self, ansi=ansi, **kw)
677
678    def __pt_repr__(self):
679        from meerschaum.utils.packages import attempt_import
680        prompt_toolkit_formatted_text = attempt_import('prompt_toolkit.formatted_text', lazy=False)
681        return prompt_toolkit_formatted_text.ANSI(pipe_repr(self, ansi=True))
682
683    def __getstate__(self) -> Dict[str, Any]:
684        """
685        Define the state dictionary (pickling).
686        """
687        return {
688            'connector_keys': self.connector_keys,
689            'metric_key': self.metric_key,
690            'location_key': self.location_key,
691            'parameters': self._attributes.get('parameters', None),
692            'instance_keys': self.instance_keys,
693        }
694
695    def __setstate__(self, _state: Dict[str, Any]):
696        """
697        Read the state (unpickling).
698        """
699        self.__init__(**_state)
700
701    def __getitem__(self, key: str) -> Any:
702        """
703        Index the pipe's attributes.
704        If the `key` cannot be found`, return `None`.
705        """
706        if key in self.attributes:
707            return self.attributes.get(key, None)
708
709        aliases = {
710            'connector': 'connector_keys',
711            'connector_key': 'connector_keys',
712            'metric': 'metric_key',
713            'location': 'location_key',
714        }
715        aliased_key = aliases.get(key, None)
716        if aliased_key is not None:
717            return self.attributes.get(aliased_key, None)
718
719        property_aliases = {
720            'instance': 'instance_keys',
721            'instance_key': 'instance_keys',
722        }
723        aliased_key = property_aliases.get(key, None)
724        if aliased_key is not None:
725            key = aliased_key
726        return getattr(self, key, None)
727
728    def __copy__(self):
729        """
730        Return a shallow copy of the current pipe.
731        """
732        return mrsm.Pipe(
733            self.connector_keys, self.metric_key, self.location_key,
734            instance=self.instance_keys,
735            parameters=self._attributes.get('parameters', None),
736        )
737
738    def __deepcopy__(self, memo):
739        """
740        Return a deep copy of the current pipe.
741        """
742        return self.__copy__()

Access Meerschaum pipes via Pipe objects.

Pipes are identified by the following:

  1. Connector keys (e.g. 'sql:main')
  2. Metric key (e.g. 'weather')
  3. Location (optional; e.g. None)

A pipe's connector keys correspond to a data source, and when the pipe is synced, its fetch definition is evaluated and executed to produce new data.

Alternatively, new data may be directly synced via pipe.sync():

>>> from meerschaum import Pipe
>>> pipe = Pipe('csv', 'weather')
>>>
>>> import pandas as pd
>>> df = pd.read_csv('weather.csv')
>>> pipe.sync(df)
Pipe( connector: str = '', metric: str = '', location: str | None = None, parameters: Dict[str, Any] | None = None, columns: Dict[str, str] | List[str] | None = None, indices: Dict[str, str | List[str]] | None = None, tags: List[str] | None = None, target: str | None = None, dtypes: Dict[str, str] | None = None, instance: str | InstanceConnector | None = None, upsert: bool | None = None, autoincrement: bool | None = None, autotime: bool | None = None, precision: str | Dict[str, str | int] | None = None, static: bool | None = None, enforce: bool | None = None, null_indices: bool | None = None, mixed_numerics: bool | None = None, compress: bool | Dict[str, Any] | None = None, temporary: bool = False, cache: bool | None = None, cache_connector_keys: str | None = None, references: List[str | Dict[str, Any] | Pipe | None] | None = None, reference: str | Dict[str, Any] | Pipe | None = None, parents: List[str | Dict[str, Any] | Pipe | None] | None = None, parent: str | Dict[str, Any] | Pipe | None = None, children: List[str | Dict[str, Any] | Pipe | None] | None = None, child: str | Dict[str, Any] | Pipe | None = None, mrsm_instance: str | InstanceConnector | None = None, connector_keys: str | None = None, metric_key: str | None = None, location_key: str | None = None, instance_keys: str | None = None, indexes: Dict[str, str] | List[str] | None = None, hypercore: bool | None = None, debug: bool = False) 
210    def __init__(
211        self,
212        connector: str = '',
213        metric: str = '',
214        location: Optional[str] = None,
215        parameters: Optional[Dict[str, Any]] = None,
216        columns: Union[Dict[str, str], List[str], None] = None,
217        indices: Optional[Dict[str, Union[str, List[str]]]] = None,
218        tags: Optional[List[str]] = None,
219        target: Optional[str] = None,
220        dtypes: Optional[Dict[str, str]] = None,
221        instance: Optional[Union[str, InstanceConnector]] = None,
222        upsert: Optional[bool] = None,
223        autoincrement: Optional[bool] = None,
224        autotime: Optional[bool] = None,
225        precision: Union[str, Dict[str, Union[str, int]], None] = None,
226        static: Optional[bool] = None,
227        enforce: Optional[bool] = None,
228        null_indices: Optional[bool] = None,
229        mixed_numerics: Optional[bool] = None,
230        compress: Union[bool, Dict[str, Any], None] = None,
231        temporary: bool = False,
232        cache: Optional[bool] = None,
233        cache_connector_keys: Optional[str] = None,
234        references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
235        reference: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
236        parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
237        parent: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
238        children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
239        child: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
240        mrsm_instance: Optional[Union[str, InstanceConnector]] = None,
241        connector_keys: Optional[str] = None,
242        metric_key: Optional[str] = None,
243        location_key: Optional[str] = None,
244        instance_keys: Optional[str] = None,
245        indexes: Union[Dict[str, str], List[str], None] = None,
246        hypercore: Optional[bool] = None,
247        debug: bool = False,
248    ):
249        """
250        Parameters
251        ----------
252        connector: str
253            Keys for the pipe's source connector, e.g. `'sql:main'`.
254
255        metric: str
256            Label for the pipe's contents, e.g. `'weather'`.
257
258        location: str, default None
259            Label for the pipe's location. Defaults to `None`.
260
261        parameters: Optional[Dict[str, Any]], default None
262            Optionally set a pipe's parameters from the constructor,
263            e.g. columns and other attributes.
264            You can edit these parameters with `edit pipes`.
265
266        columns: Union[Dict[str, str], List[str], None], default None
267            Set the `columns` dictionary of `parameters`.
268            If `parameters` is also provided, this dictionary is added under the `'columns'` key.
269
270        indices: Optional[Dict[str, Union[str, List[str]]]], default None
271            Set the `indices` dictionary of `parameters`.
272            If `parameters` is also provided, this dictionary is added under the `'indices'` key.
273
274        tags: Optional[List[str]], default None
275            A list of strings to be added under the `'tags'` key of `parameters`.
276            You can select pipes with certain tags using `--tags`.
277
278        dtypes: Optional[Dict[str, str]], default None
279            Set the `dtypes` dictionary of `parameters`.
280            If `parameters` is also provided, this dictionary is added under the `'dtypes'` key.
281
282        mrsm_instance: Optional[Union[str, InstanceConnector]], default None
283            Connector for the Meerschaum instance where the pipe resides.
284            Defaults to the preconfigured default instance (`'sql:main'`).
285
286        instance: Optional[Union[str, InstanceConnector]], default None
287            Alias for `mrsm_instance`. If `mrsm_instance` is supplied, this value is ignored.
288
289        upsert: Optional[bool], default None
290            If `True`, set `upsert` to `True` in the parameters.
291
292        autoincrement: Optional[bool], default None
293            If `True`, set `autoincrement` in the parameters.
294
295        autotime: Optional[bool], default None
296            If `True`, set `autotime` in the parameters.
297
298        precision: Union[str, Dict[str, Union[str, int]], None], default None
299            If provided, set `precision` in the parameters.
300            This may be either a string (the precision unit) or a dictionary of in the form
301            `{'unit': <unit>, 'interval': <interval>}`.
302            Default is determined by the `datetime` column dtype
303            (e.g. `datetime64[us]` is `microsecond` precision).
304
305        static: Optional[bool], default None
306            If `True`, set `static` in the parameters.
307
308        enforce: Optional[bool], default None
309            If `False`, skip data type enforcement.
310            Default behavior is `True`.
311
312        null_indices: Optional[bool], default None
313            Set to `False` if there will be no null values in the index columns.
314            Defaults to `True`.
315
316        mixed_numerics: bool, default None
317            If `True`, integer columns will be converted to `numeric` when floats are synced.
318            Set to `False` to disable this behavior.
319            Defaults to `True`.
320
321        compress: Union[bool, Dict[str, Any], None], default None
322            If `True` (or a dictionary of compression settings), mark the pipe for compression.
323            For TimescaleDB hypertables, a columnstore (compression) policy is installed
324            automatically on sync. A dictionary may override `segmentby`, `orderby`, and `after`.
325            Defaults to `False`.
326
327        hypercore: Optional[bool], default None
328            Set `hypercore` in the parameters. For TimescaleDB hypertables, the Hypercore
329            columnstore is enabled at table creation by default (declaring
330            `segmentby`/`orderby` in `CREATE TABLE`), which causes TimescaleDB to auto-create a
331            columnstore policy. Set to `False` for a plain row-store hypertable.
332            Has no effect unless the pipe is a hypertable (`hypertable`, default `True`).
333
334        temporary: bool, default False
335            If `True`, prevent instance tables (pipes, users, plugins) from being created.
336
337        cache: Optional[bool], default None
338            If `True`, cache the pipe's metadata to disk (in addition to in-memory caching).
339            If `cache` is not explicitly `True`, it is set to `False` if `temporary` is `True`.
340            Defaults to `True` (from `None`).
341
342        cache_connector_keys: Optional[str], default None
343            If provided, use the keys to a Valkey connector (e.g. `valkey:main`).
344
345        references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
346            If provided, inherit the parameters of the reference Pipe(s).
347            May be equal to a string of the Pipe constructor, a dictionary of constructor keys,
348            a Pipe itself, or a list of any of these values.
349
350        parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
351            Set references for parent pipes. See `references` for values.
352
353        children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
354            Set references for child pipes. See `references` for values.
355
356        """
357        from meerschaum.utils.warnings import error, warn
358        if (not connector and not connector_keys) or (not metric and not metric_key):
359            error(
360                "Please provide strings for the connector and metric\n    "
361                + "(first two positional arguments)."
362            )
363
364        ### Fall back to legacy `location_key` just in case.
365        if not location:
366            location = location_key
367
368        if not connector:
369            connector = connector_keys
370
371        if not metric:
372            metric = metric_key
373
374        if location in ('[None]', 'None'):
375            location = None
376
377        from meerschaum._internal.static import STATIC_CONFIG
378        negation_prefix = STATIC_CONFIG['system']['fetch_pipes_keys']['negation_prefix']
379        for k in (connector, metric, location, *(tags or [])):
380            if str(k).startswith(negation_prefix):
381                error(f"A pipe's keys and tags cannot start with the prefix '{negation_prefix}'.")
382
383        self._connector_keys = str(connector)
384        self._connector_key = self._connector_keys ### Alias
385        self._metric_key = metric
386        self._location_key = location
387        self.temporary = temporary
388        self.cache = (
389            cache
390            if cache is not None
391            else ((not temporary) and get_config('pipes', 'cache', 'enabled', warn=False))
392        )
393        self.cache_connector_keys = (
394            str(cache_connector_keys)
395            if cache_connector_keys is not None
396            else None
397        )
398        self.debug = debug
399
400        _mrsm_instance = mrsm_instance if mrsm_instance is not None else (instance or instance_keys)
401        if _mrsm_instance is None:
402            _mrsm_instance = get_config('meerschaum', 'instance', patch=True)
403
404        if not isinstance(_mrsm_instance, str):
405            self._instance_connector = _mrsm_instance
406            self._instance_keys = str(_mrsm_instance)
407        else:
408            self._instance_keys = _mrsm_instance
409
410        if self._instance_keys == 'sql:memory':
411            self.cache = False
412
413        from meerschaum import _core
414        from ._live_parameters import Attrs, new_raw_root
415        ### Keys and raw parameters live in a `_core.Pipe`; `_raw_root` is a read-through
416        ### view of its raw map. Constructing never touches the instance.
417        self._core_pipe = _core.Pipe(
418            self._connector_keys,
419            self._metric_key,
420            self._location_key,
421            self._instance_keys,
422            None,
423        )
424        self._raw_root = new_raw_root(self._core_pipe)
425        self._attributes = Attrs(
426            self,
427            {
428                'connector_keys': self._connector_keys,
429                'metric_key': self._metric_key,
430                'location_key': self._location_key,
431                'parameters': {},
432            },
433            self._raw_root,
434        )
435
436        ### only set parameters if values are provided
437        if isinstance(parameters, dict):
438            self._attributes['parameters'] = parameters
439        else:
440            if parameters is not None:
441                warn(f"The provided parameters are of invalid type '{type(parameters)}'.")
442            self._attributes['parameters'] = {}
443
444        columns = columns or self._attributes.get('parameters', {}).get('columns', None)
445        if isinstance(columns, (list, tuple)):
446            columns = {str(col): str(col) for col in columns}
447        if isinstance(columns, dict):
448            self._attributes['parameters']['columns'] = columns
449        elif isinstance(columns, str) and 'Pipe(' in columns:
450            pass
451        elif columns is not None:
452            warn(f"The provided columns are of invalid type '{type(columns)}'.")
453
454        indices = (
455            indices
456            or indexes
457            or self._attributes.get('parameters', {}).get('indices', None)
458            or self._attributes.get('parameters', {}).get('indexes', None)
459        )
460        if isinstance(indices, dict):
461            indices_key = (
462                'indexes'
463                if 'indexes' in self._attributes['parameters']
464                else 'indices'
465            )
466            self._attributes['parameters'][indices_key] = indices
467
468        if isinstance(tags, (list, tuple)):
469            self._attributes['parameters']['tags'] = tags
470        elif tags is not None:
471            warn(f"The provided tags are of invalid type '{type(tags)}'.")
472
473        if isinstance(target, str):
474            self._attributes['parameters']['target'] = target
475        elif target is not None:
476            warn(f"The provided target is of invalid type '{type(target)}'.")
477
478        if isinstance(dtypes, dict):
479            self._attributes['parameters']['dtypes'] = dtypes
480        elif dtypes is not None:
481            warn(f"The provided dtypes are of invalid type '{type(dtypes)}'.")
482
483        if isinstance(upsert, bool):
484            self._attributes['parameters']['upsert'] = upsert
485
486        if isinstance(autoincrement, bool):
487            self._attributes['parameters']['autoincrement'] = autoincrement
488
489        if isinstance(autotime, bool):
490            self._attributes['parameters']['autotime'] = autotime
491
492        if isinstance(precision, dict):
493            self._attributes['parameters']['precision'] = precision
494        elif isinstance(precision, str):
495            self._attributes['parameters']['precision'] = {'unit': precision}
496
497        if isinstance(static, bool):
498            self._attributes['parameters']['static'] = static
499            self._static = static
500
501        if isinstance(enforce, bool):
502            self._attributes['parameters']['enforce'] = enforce
503
504        if isinstance(null_indices, bool):
505            self._attributes['parameters']['null_indices'] = null_indices
506
507        if isinstance(mixed_numerics, bool):
508            self._attributes['parameters']['mixed_numerics'] = mixed_numerics
509
510        if isinstance(compress, (bool, dict)):
511            self._attributes['parameters']['compress'] = compress
512
513        if isinstance(hypercore, bool):
514            self._attributes['parameters']['hypercore'] = hypercore
515
516        self._cache_locks = collections.defaultdict(lambda: threading.RLock())
517        self._attributes_lock = threading.RLock()
518
519        if references is not None or reference is not None:
520            reference_vals = references if references is not None else reference
521            self.references = reference_vals
522
523        if parents is not None or parent is not None:
524            parent_vals = parents if parents is not None else parent
525            self.parents = parent_vals
526
527        if children is not None or child is not None:
528            children_vals = children if children is not None else child
529            self.children = children_vals
Parameters
  • connector (str): Keys for the pipe's source connector, e.g. 'sql:main'.
  • metric (str): Label for the pipe's contents, e.g. 'weather'.
  • location (str, default None): Label for the pipe's location. Defaults to None.
  • parameters (Optional[Dict[str, Any]], default None): Optionally set a pipe's parameters from the constructor, e.g. columns and other attributes. You can edit these parameters with edit pipes.
  • columns (Union[Dict[str, str], List[str], None], default None): Set the columns dictionary of parameters. If parameters is also provided, this dictionary is added under the 'columns' key.
  • indices (Optional[Dict[str, Union[str, List[str]]]], default None): Set the indices dictionary of parameters. If parameters is also provided, this dictionary is added under the 'indices' key.
  • tags (Optional[List[str]], default None): A list of strings to be added under the 'tags' key of parameters. You can select pipes with certain tags using --tags.
  • dtypes (Optional[Dict[str, str]], default None): Set the dtypes dictionary of parameters. If parameters is also provided, this dictionary is added under the 'dtypes' key.
  • mrsm_instance (Optional[Union[str, InstanceConnector]], default None): Connector for the Meerschaum instance where the pipe resides. Defaults to the preconfigured default instance ('sql:main').
  • instance (Optional[Union[str, InstanceConnector]], default None): Alias for mrsm_instance. If mrsm_instance is supplied, this value is ignored.
  • upsert (Optional[bool], default None): If True, set upsert to True in the parameters.
  • autoincrement (Optional[bool], default None): If True, set autoincrement in the parameters.
  • autotime (Optional[bool], default None): If True, set autotime in the parameters.
  • precision (Union[str, Dict[str, Union[str, int]], None], default None): If provided, set precision in the parameters. This may be either a string (the precision unit) or a dictionary of in the form {'unit': <unit>, 'interval': <interval>}. Default is determined by the datetime column dtype (e.g. datetime64[us] is microsecond precision).
  • static (Optional[bool], default None): If True, set static in the parameters.
  • enforce (Optional[bool], default None): If False, skip data type enforcement. Default behavior is True.
  • null_indices (Optional[bool], default None): Set to False if there will be no null values in the index columns. Defaults to True.
  • mixed_numerics (bool, default None): If True, integer columns will be converted to numeric when floats are synced. Set to False to disable this behavior. Defaults to True.
  • compress (Union[bool, Dict[str, Any], None], default None): If True (or a dictionary of compression settings), mark the pipe for compression. For TimescaleDB hypertables, a columnstore (compression) policy is installed automatically on sync. A dictionary may override segmentby, orderby, and after. Defaults to False.
  • hypercore (Optional[bool], default None): Set hypercore in the parameters. For TimescaleDB hypertables, the Hypercore columnstore is enabled at table creation by default (declaring segmentby/orderby in CREATE TABLE), which causes TimescaleDB to auto-create a columnstore policy. Set to False for a plain row-store hypertable. Has no effect unless the pipe is a hypertable (hypertable, default True).
  • temporary (bool, default False): If True, prevent instance tables (pipes, users, plugins) from being created.
  • cache (Optional[bool], default None): If True, cache the pipe's metadata to disk (in addition to in-memory caching). If cache is not explicitly True, it is set to False if temporary is True. Defaults to True (from None).
  • cache_connector_keys (Optional[str], default None): If provided, use the keys to a Valkey connector (e.g. valkey:main).
  • references (Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None): If provided, inherit the parameters of the reference Pipe(s). May be equal to a string of the Pipe constructor, a dictionary of constructor keys, a Pipe itself, or a list of any of these values.
  • parents (Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None): Set references for parent pipes. See references for values.
  • children (Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None): Set references for child pipes. See references for values.
temporary
cache
cache_connector_keys
debug
metric_key: str 
531    @property
532    def metric_key(self) -> str:
533        """
534        Return the pipe's metric key.
535        """
536        return self._core_pipe.metric_key

Return the pipe's metric key.

metric: str 
538    @property
539    def metric(self) -> str:
540        """
541        Return the pipe's metric key.
542        """
543        return self.metric_key

Return the pipe's metric key.

location_key: str | None 
545    @property
546    def location_key(self) -> Union[str, None]:
547        """
548        Return the pipe's location key.
549        """
550        return self._core_pipe.location_key

Return the pipe's location key.

location: str | None 
552    @property
553    def location(self) -> Union[str, None]:
554        """
555        Return the pipe's location key.
556        """
557        return self.location_key

Return the pipe's location key.

meta 
559    @property
560    def meta(self):
561        """
562        Return the four keys needed to reconstruct this pipe.
563        """
564        return {
565            'connector_keys': self.connector_keys,
566            'metric_key': self.metric_key,
567            'location_key': self.location_key,
568            'instance_keys': self.instance_keys,
569        }

Return the four keys needed to reconstruct this pipe.

def keys(self) -> List[str]: 
571    def keys(self) -> List[str]:
572        """
573        Return the ordered keys for this pipe.
574        """
575        return {
576            key: val
577            for key, val in self.meta.items()
578            if key != 'instance'
579        }

Return the ordered keys for this pipe.

instance_keys: str 
581    @property
582    def instance_keys(self) -> str:
583        """
584        Return the pipe's instance keys.
585        """
586        return self._core_pipe.instance_keys

Return the pipe's instance keys.

instance: InstanceConnector | str 
588    @property
589    def instance(self) -> Union[InstanceConnector, str]:
590        """
591        Return the pipe's instance connector or keys.
592        """
593        conn = self.instance_connector
594        if conn is None:
595            return self.instance_keys
596        return conn

Return the pipe's instance connector or keys.

instance_connector: InstanceConnector | None 
598    @property
599    def instance_connector(self) -> Union[InstanceConnector, None]:
600        """
601        The instance connector on which this pipe resides.
602        """
603        if '_instance_connector' not in self.__dict__:
604            from meerschaum.connectors.parse import parse_instance_keys
605            conn = parse_instance_keys(self.instance_keys)
606            if conn:
607                self._instance_connector = conn
608            else:
609                return None
610        return self._instance_connector

The instance connector on which this pipe resides.

connector_keys: str 
612    @property
613    def connector_keys(self) -> str:
614        """
615        Return the pipe's connector keys.
616        """
617        return self._core_pipe.connector_keys

Return the pipe's connector keys.

connector_key: str 
619    @property
620    def connector_key(self) -> str:
621        """
622        Legacy: use `Pipe.connector_keys` instead.
623        """
624        return self.connector_keys

Legacy: use Pipe.connector_keys instead.

connector: "Union['Connector', str]" 
626    @property
627    def connector(self) -> Union['Connector', str]:
628        """
629        The connector to the data source.
630        """
631        if '_connector' not in self.__dict__:
632            from meerschaum.connectors.parse import parse_instance_keys
633            import warnings
634            with warnings.catch_warnings():
635                warnings.simplefilter('ignore')
636                try:
637                    conn = parse_instance_keys(self.connector_keys)
638                except Exception:
639                    conn = None
640            if conn:
641                self._connector = conn
642            else:
643                return self._connector_keys
644        return self._connector

The connector to the data source.

def fetch( self, begin: datetime.datetime | int | str | None = '', end: datetime.datetime | int | None = None, check_existing: bool = True, sync_chunks: bool = False, debug: bool = False, as_polars: bool | None = None, as_pandas: bool | None = None, **kw: Any) -> "Union['pl.DataFrame', 'pd.DataFrame', Iterator[Any], None]": 
 21def fetch(
 22    self,
 23    begin: Union[datetime, int, str, None] = '',
 24    end: Union[datetime, int, None] = None,
 25    check_existing: bool = True,
 26    sync_chunks: bool = False,
 27    debug: bool = False,
 28    as_polars: Optional[bool] = None,
 29    as_pandas: Optional[bool] = None,
 30    **kw: Any
 31) -> Union['pl.DataFrame', 'pd.DataFrame', Iterator[Any], None]:
 32    """
 33    Fetch a Pipe's latest data from its connector.
 34
 35    Parameters
 36    ----------
 37    begin: Union[datetime, str, None], default '':
 38        If provided, only fetch data newer than or equal to `begin`.
 39
 40    end: Optional[datetime], default None:
 41        If provided, only fetch data older than or equal to `end`.
 42
 43    check_existing: bool, default True
 44        If `False`, do not apply the backtrack interval.
 45
 46    sync_chunks: bool, default False
 47        If `True` and the pipe's connector is of type `'sql'`, begin syncing chunks while fetching
 48        loads chunks into memory.
 49
 50    debug: bool, default False
 51        Verbosity toggle.
 52
 53    as_polars: Optional[bool], default None
 54        `True` returns Polars frames and `False` pandas ones, whatever `meerschaum:dataframe`
 55        says (see `as_pandas`).
 56
 57    as_pandas: Optional[bool], default None
 58        `True` returns pandas frames and `False` Polars ones. By default (`None`) the frames are
 59        the library of the `meerschaum:dataframe` setting, whatever the source returned.
 60        A `chunk_hook` is given chunks of the same library.
 61
 62    Returns
 63    -------
 64    A DataFrame of the newest unseen data (Polars or pandas, per `meerschaum:dataframe`), or a
 65    generator of chunks for a SQL source.
 66
 67    """
 68    from meerschaum import _core
 69    from meerschaum.core.Pipe._data import _rust_context, _wants_polars
 70    polars = _wants_polars(False, as_polars, as_pandas)
 71    try:
 72        context = _rust_context(self, debug=debug)
 73    except Exception as e:
 74        ### v4 checked the source before it touched the instance: a source without `fetch()`
 75        ### answers `None` with a warning even when the instance cannot be parsed.
 76        if type(e) is Exception and 'fetch' not in dir(self.connector):
 77            from meerschaum.utils.warnings import warn
 78            warn(f"No `fetch()` function defined for connector '{self.connector}'")
 79            return None
 80        raise
 81    begin, end = self.parse_date_bounds(begin, end)
 82    fetched = _core.pipe_fetch(
 83        self,
 84        context,
 85        begin=begin,
 86        end=end,
 87        check_existing=check_existing,
 88        params=kw.get('params', None),
 89        kwargs={
 90            key: val
 91            for key, val in kw.items()
 92            if key not in ('params', 'chunk_hook')
 93        },
 94        as_polars=polars,
 95        sync_chunks=sync_chunks,
 96        chunk_hook=kw.get('chunk_hook', None),
 97        debug=debug,
 98    )
 99    ### A SQL source's rows come in chunks, as a generator.
100    return (chunk for chunk in fetched) if hasattr(fetched, '__next__') else fetched

Fetch a Pipe's latest data from its connector.

Parameters
  • begin (Union[datetime, str, None], default '':): If provided, only fetch data newer than or equal to begin.
  • end (Optional[datetime], default None:): If provided, only fetch data older than or equal to end.
  • check_existing (bool, default True): If False, do not apply the backtrack interval.
  • sync_chunks (bool, default False): If True and the pipe's connector is of type 'sql', begin syncing chunks while fetching loads chunks into memory.
  • debug (bool, default False): Verbosity toggle.
  • as_polars (Optional[bool], default None): True returns Polars frames and False pandas ones, whatever meerschaum:dataframe says (see as_pandas).
  • as_pandas (Optional[bool], default None): True returns pandas frames and False Polars ones. By default (None) the frames are the library of the meerschaum:dataframe setting, whatever the source returned. A chunk_hook is given chunks of the same library.
Returns

  • **A DataFrame of the newest unseen data (Polars or pandas, per meerschaum** (dataframe), or a):

  • generator of chunks for a SQL source.

def get_backtrack_interval( self, check_existing: bool = True, debug: bool = False) -> datetime.timedelta | int: 
103def get_backtrack_interval(
104    self,
105    check_existing: bool = True,
106    debug: bool = False,
107) -> Union[timedelta, int]:
108    """
109    Get the chunk interval to use for this pipe.
110
111    Parameters
112    ----------
113    check_existing: bool, default True
114        If `False`, return a backtrack_interval of 0 minutes.
115
116    Returns
117    -------
118    The backtrack interval (`timedelta` or `int`) to use with this pipe's `datetime` axis.
119    """
120    from meerschaum import _core
121    return _core.pipe_get_backtrack_interval(self, check_existing, self.dtypes)

Get the chunk interval to use for this pipe.

Parameters
  • check_existing (bool, default True): If False, return a backtrack_interval of 0 minutes.
Returns
  • The backtrack interval (timedelta or int) to use with this pipe's datetime axis.
def get_data( self, select_columns: List[str] | None = None, omit_columns: List[str] | None = None, begin: datetime.datetime | int | str | None = None, end: datetime.datetime | int | str | None = None, params: Dict[str, Any] | None = None, as_docs: bool = False, as_iterator: bool = False, as_chunks: bool = False, as_dask: bool = False, as_polars: bool | None = None, add_missing_columns: bool = False, chunk_interval: datetime.timedelta | int | None = None, order: str | None = 'asc', limit: int | None = None, fresh: bool = False, silent: bool = False, debug: bool = False, as_pandas: bool | None = None, **kw: Any) -> "Union['pl.DataFrame', 'pd.DataFrame', Iterator[Any], List[Dict[str, Any]], None]": 
 85def get_data(
 86    self,
 87    select_columns: Optional[List[str]] = None,
 88    omit_columns: Optional[List[str]] = None,
 89    begin: Union[datetime, int, str, None] = None,
 90    end: Union[datetime, int, str, None] = None,
 91    params: Optional[Dict[str, Any]] = None,
 92    as_docs: bool = False,
 93    as_iterator: bool = False,
 94    as_chunks: bool = False,
 95    as_dask: bool = False,
 96    as_polars: Optional[bool] = None,
 97    add_missing_columns: bool = False,
 98    chunk_interval: Union[timedelta, int, None] = None,
 99    order: Optional[str] = 'asc',
100    limit: Optional[int] = None,
101    fresh: bool = False,
102    silent: bool = False,
103    debug: bool = False,
104    as_pandas: Optional[bool] = None,
105    **kw: Any
106) -> Union['pl.DataFrame', 'pd.DataFrame', Iterator[Any], List[Dict[str, Any]], None]:
107    """
108    Get a pipe's data from the instance connector.
109
110    Parameters
111    ----------
112    select_columns: Optional[List[str]], default None
113        If provided, only select these given columns.
114        Otherwise select all available columns (i.e. `SELECT *`).
115
116    omit_columns: Optional[List[str]], default None
117        If provided, remove these columns from the selection.
118
119    begin: Union[datetime, int, str, None], default None
120        Lower bound datetime to begin searching for data (inclusive).
121        Translates to a `WHERE` clause like `WHERE datetime >= begin`.
122        Defaults to `None`.
123
124    end: Union[datetime, int, str, None], default None
125        Upper bound datetime to stop searching for data (exclusive).
126        Translates to a `WHERE` clause like `WHERE datetime < end`.
127        Defaults to `None`.
128
129    params: Optional[Dict[str, Any]], default None
130        Filter the rows by column values (a `WHERE` clause), e.g.
131        `params={'station': 'KATL'}`:
132
133        - a value is `=`: `{'station': 'KATL'}`;
134        - a list is `IN`: `{'station': ['KATL', 'KGMU']}`;
135        - a leading `_` negates: `{'station': '_KATL'}` is `!=`, `{'station': ['_KATL']}`
136          is `NOT IN`;
137        - `None` (or `'None'`) is `IS NULL` and `'_None'` is `IS NOT NULL`;
138        - a dictionary value is compared as JSON text.
139
140        See `meerschaum.utils.sql.build_where` for more details.
141
142    as_docs: bool, default False
143        If `True`, return a list of dictionaries rather than a DataFrame.
144        Relies on `get_pipe_docs` from the instance connector if implemented.
145        May be combined with `as_chunks` to return an `Iterator[List[Dict]]`
146        chunked by time bounds (useful for large result sets without pandas overhead).
147
148    as_iterator: bool, default False
149        If `True`, return a generator of chunks of pipe data.
150        When combined with `as_docs=True`, yields `List[Dict]` per chunk instead of DataFrames.
151
152    as_chunks: bool, default False
153        Alias for `as_iterator`.
154        When combined with `as_docs=True`, yields `List[Dict]` per chunk instead of DataFrames.
155
156    as_dask: bool, default False
157        Not supported in Meerschaum 5: `True` raises a `TypeError`.
158        Read the data in chunks with `as_iterator=True` (or as one Polars frame) instead.
159
160    as_polars: Optional[bool], default None
161        `True` returns a `polars.DataFrame` and `False` a pandas one, whatever the
162        `meerschaum:dataframe` setting says (see `as_pandas`). Chunks (`as_iterator=True`)
163        follow it too. This may not be combined with `as_docs`.
164
165    add_missing_columns: bool, default False
166        If `True`, add any missing columns from `Pipe.dtypes` to the dataframe.
167
168    chunk_interval: Union[timedelta, int, None], default None
169        If `as_iterator`, then return chunks with `begin` and `end` separated by this interval.
170        This may be set under `pipe.parameters['chunk_minutes']`.
171        By default, use a timedelta of 43200 minutes (30 days).
172        If `chunk_interval` is an integer and the `datetime` axis a timestamp,
173        then use a timedelta with the number of minutes configured to this value.
174        If the `datetime` axis is an integer, an `int` is used verbatim as the number of
175        epoch units, and a `timedelta` is scaled into epoch units by the pipe's `precision`.
176
177    order: Optional[str], default 'asc'
178        If `order` is not `None`, sort the resulting dataframe by indices.
179
180    limit: Optional[int], default None
181        If provided, cap the dataframe to this many rows.
182
183    fresh: bool, default False
184        If `True`, skip local cache and directly query the instance connector.
185
186    silent: bool, default False
187        If `True`, return `None` instead of raising when the instance cannot be read
188        (Meerschaum 4's behavior). By default, database errors raise; a pipe whose table
189        does not exist yet returns `None` either way.
190
191    debug: bool, default False
192        Verbosity toggle.
193        Defaults to `False`.
194
195    as_pandas: Optional[bool], default None
196        `True` returns a pandas DataFrame (a `geopandas.GeoDataFrame` when the pipe has
197        geometry columns) and `False` a Polars one. By default (`None`) the frame is the
198        library of the `meerschaum:dataframe` setting (`MRSM_DATAFRAME` overrides it):
199        Polars for a new root, pandas for a root created before Meerschaum 5.0.0rc15.
200        Polars frames carry geometry as WKB (binary) with the CRS in the field metadata.
201
202    kw: Any
203        Other keyword arguments are passed on to the instance connector (a SQL instance
204        reads `begin_add_minutes`, `end_add_minutes`, `replace_nulls` and
205        `skip_existing_cols_check`; a custom instance connector may read its own). A keyword
206        that nothing reads is ignored with a warning that suggests the parameter you probably
207        meant (`start` → `begin`, `where` or `filters` → `params`).
208
209    Returns
210    -------
211    A DataFrame of the pipe's data: Polars or pandas, per `meerschaum:dataframe`
212    (or `as_polars` / `as_pandas`).
213    A `List[Dict]` if `as_docs=True`.
214    An iterator of DataFrames if `as_chunks=True` (or `as_iterator=True`).
215    An `Iterator[List[Dict]]` if both `as_docs=True` and `as_chunks=True`.
216    `None` if the pipe's table does not exist yet.
217
218    """
219    from meerschaum import _core
220    from meerschaum.core.Pipe._keywords import warn_unknown_get_data_keywords
221    warn_unknown_get_data_keywords(self, kw, as_docs=as_docs)
222    if as_polars and (as_docs or as_dask):
223        raise ValueError("`as_polars` may not be combined with `as_docs` or `as_dask`.")
224    if as_pandas and (as_docs or as_dask):
225        raise ValueError("`as_pandas` may not be combined with `as_docs` or `as_dask`.")
226    if as_dask:
227        raise TypeError(
228            "Dask is not supported in Meerschaum 5. Use `as_iterator=True` for chunks "
229            "or `as_polars=True`. See the migration guide: reference/migrating-to-v5."
230        )
231    polars = _wants_polars(as_docs, as_polars, as_pandas)
232    try:
233        context = _rust_context(self, debug=debug)
234        begin, end = self.parse_date_bounds(begin, end, debug=debug)
235        if as_iterator or as_chunks:
236            chunks = _iterate_chunks(_core.pipe_get_data_chunks(
237                self,
238                context,
239                select_columns=select_columns,
240                omit_columns=omit_columns,
241                begin=begin,
242                end=end,
243                params=params,
244                as_docs=as_docs,
245                as_polars=polars,
246                chunk_interval=chunk_interval,
247                order=order,
248                limit=limit,
249                kwargs=kw,
250                debug=debug,
251            ))
252            return _silent_chunks(chunks) if silent else chunks
253        return _core.pipe_get_data(
254            self,
255            context,
256            select_columns=select_columns,
257            omit_columns=omit_columns,
258            begin=begin,
259            end=end,
260            params=params,
261            as_docs=as_docs,
262            as_polars=polars,
263            add_missing_columns=add_missing_columns,
264            order=order,
265            limit=limit,
266            kwargs=kw,
267            debug=debug,
268        )
269    except Exception:
270        ### Database errors raise; `silent=True` keeps Meerschaum 4's `None`.
271        if silent:
272            return None
273        raise

Get a pipe's data from the instance connector.

Parameters
  • select_columns (Optional[List[str]], default None): If provided, only select these given columns. Otherwise select all available columns (i.e. SELECT *).
  • omit_columns (Optional[List[str]], default None): If provided, remove these columns from the selection.
  • begin (Union[datetime, int, str, None], default None): Lower bound datetime to begin searching for data (inclusive). Translates to a WHERE clause like WHERE datetime >= begin. Defaults to None.
  • end (Union[datetime, int, str, None], default None): Upper bound datetime to stop searching for data (exclusive). Translates to a WHERE clause like WHERE datetime < end. Defaults to None.

  • params (Optional[Dict[str, Any]], default None): Filter the rows by column values (a WHERE clause), e.g. params={'station': 'KATL'}:

    • a value is =: {'station': 'KATL'};
    • a list is IN: {'station': ['KATL', 'KGMU']};
    • a leading _ negates: {'station': '_KATL'} is !=, {'station': ['_KATL']} is NOT IN;
    • None (or 'None') is IS NULL and '_None' is IS NOT NULL;
    • a dictionary value is compared as JSON text.

    See meerschaum.utils.sql.build_where for more details.

  • as_docs (bool, default False): If True, return a list of dictionaries rather than a DataFrame. Relies on get_pipe_docs from the instance connector if implemented. May be combined with as_chunks to return an Iterator[List[Dict]] chunked by time bounds (useful for large result sets without pandas overhead).
  • as_iterator (bool, default False): If True, return a generator of chunks of pipe data. When combined with as_docs=True, yields List[Dict] per chunk instead of DataFrames.
  • as_chunks (bool, default False): Alias for as_iterator. When combined with as_docs=True, yields List[Dict] per chunk instead of DataFrames.
  • as_dask (bool, default False): Not supported in Meerschaum 5: True raises a TypeError. Read the data in chunks with as_iterator=True (or as one Polars frame) instead.
  • as_polars (Optional[bool], default None): True returns a polars.DataFrame and False a pandas one, whatever the meerschaum:dataframe setting says (see as_pandas). Chunks (as_iterator=True) follow it too. This may not be combined with as_docs.
  • add_missing_columns (bool, default False): If True, add any missing columns from Pipe.dtypes to the dataframe.
  • chunk_interval (Union[timedelta, int, None], default None): If as_iterator, then return chunks with begin and end separated by this interval. This may be set under pipe.parameters['chunk_minutes']. By default, use a timedelta of 43200 minutes (30 days). If chunk_interval is an integer and the datetime axis a timestamp, then use a timedelta with the number of minutes configured to this value. If the datetime axis is an integer, an int is used verbatim as the number of epoch units, and a timedelta is scaled into epoch units by the pipe's precision.
  • order (Optional[str], default 'asc'): If order is not None, sort the resulting dataframe by indices.
  • limit (Optional[int], default None): If provided, cap the dataframe to this many rows.
  • fresh (bool, default False): If True, skip local cache and directly query the instance connector.
  • silent (bool, default False): If True, return None instead of raising when the instance cannot be read (Meerschaum 4's behavior). By default, database errors raise; a pipe whose table does not exist yet returns None either way.
  • debug (bool, default False): Verbosity toggle. Defaults to False.
  • as_pandas (Optional[bool], default None): True returns a pandas DataFrame (a geopandas.GeoDataFrame when the pipe has geometry columns) and False a Polars one. By default (None) the frame is the library of the meerschaum:dataframe setting (MRSM_DATAFRAME overrides it): Polars for a new root, pandas for a root created before Meerschaum 5.0.0rc15. Polars frames carry geometry as WKB (binary) with the CRS in the field metadata.
  • kw (Any): Other keyword arguments are passed on to the instance connector (a SQL instance reads begin_add_minutes, end_add_minutes, replace_nulls and skip_existing_cols_check; a custom instance connector may read its own). A keyword that nothing reads is ignored with a warning that suggests the parameter you probably meant (start → begin, where or filters → params).
Returns

  • **A DataFrame of the pipe's data: Polars or pandas, per meerschaum** (dataframe):

  • (or as_polars / as_pandas).

  • A List[Dict] if as_docs=True.
  • An iterator of DataFrames if as_chunks=True (or as_iterator=True).
  • An Iterator[List[Dict]] if both as_docs=True and as_chunks=True.
  • None if the pipe's table does not exist yet.
def get_backtrack_data( self, backtrack_minutes: int | None = None, begin: datetime.datetime | int | None = None, params: Dict[str, Any] | None = None, limit: int | None = None, fresh: bool = False, debug: bool = False, **kw: Any) -> pandas.DataFrame | None: 
276def get_backtrack_data(
277    self,
278    backtrack_minutes: Optional[int] = None,
279    begin: Union[datetime, int, None] = None,
280    params: Optional[Dict[str, Any]] = None,
281    limit: Optional[int] = None,
282    fresh: bool = False,
283    debug: bool = False,
284    **kw: Any
285) -> Optional['pd.DataFrame']:
286    """
287    Get the most recent data from the instance connector as a Pandas DataFrame.
288
289    Parameters
290    ----------
291    backtrack_minutes: Optional[int], default None
292        How many minutes from `begin` to select from.
293        If `None`, use `pipe.parameters['fetch']['backtrack_minutes']`.
294
295    begin: Optional[datetime], default None
296        The starting point to search for data.
297        If begin is `None` (default), use the most recent observed datetime
298        (AKA sync_time).
299
300        ```
301        E.g. begin = 02:00
302
303        Search this region.           Ignore this, even if there's data.
304        /  /  /  /  /  /  /  /  /  |
305        -----|----------|----------|----------|----------|----------|
306        00:00      01:00      02:00      03:00      04:00      05:00
307
308        ```
309
310    params: Optional[Dict[str, Any]], default None
311        The standard Meerschaum `params` query dictionary.
312
313    limit: Optional[int], default None
314        If provided, cap the number of rows to be returned.
315
316    fresh: bool, default False
317        If `True`, Ignore local cache and pull directly from the instance connector.
318        Only comes into effect if a pipe was created with `cache=True`.
319
320    debug: bool default False
321        Verbosity toggle.
322
323    **kw: Any
324        Also `end` (exclusive), `select_columns`, `omit_columns`, `order` (default `'desc'`),
325        `as_docs`, `as_polars`, `as_pandas` and `add_missing_columns`, as in `get_data()`.
326        Other keywords go to the instance connector; a keyword nothing reads is ignored with a
327        warning that suggests the parameter you probably meant.
328
329    Returns
330    -------
331    A DataFrame (Polars or pandas, per `meerschaum:dataframe`, as `get_data()`) of the pipe's
332    data corresponding to the provided parameters. Backtrack data is a convenient way to get a
333    pipe's data "backtracked" from the most recent datetime.
334    """
335    from meerschaum import _core
336    from meerschaum.core.Pipe._keywords import BACKTRACK_KEYWORDS, warn_unknown_backtrack_keywords
337    warn_unknown_backtrack_keywords(self, kw)
338    polars = _wants_polars(
339        kw.get('as_docs', False),
340        kw.get('as_polars', None),
341        kw.get('as_pandas', None),
342    )
343    context = _rust_context(self, debug=debug)
344    named = BACKTRACK_KEYWORDS
345    end = kw.get('end', None)
346    return _core.pipe_get_backtrack_data(
347        self,
348        context,
349        backtrack_minutes=backtrack_minutes,
350        begin=self.parse_date_bounds(begin, debug=debug),
351        params=params,
352        limit=limit,
353        order=kw.get('order', 'desc') or 'desc',
354        select_columns=kw.get('select_columns', None),
355        omit_columns=kw.get('omit_columns', None),
356        end=self.parse_date_bounds(end, debug=debug) if end is not None else None,
357        as_docs=kw.get('as_docs', False),
358        as_polars=polars,
359        add_missing_columns=kw.get('add_missing_columns', False),
360        kwargs={key: val for key, val in kw.items() if key not in named},
361        debug=debug,
362    )

Get the most recent data from the instance connector as a Pandas DataFrame.

Parameters
  • backtrack_minutes (Optional[int], default None): How many minutes from begin to select from. If None, use pipe.parameters['fetch']['backtrack_minutes'].

  • begin (Optional[datetime], default None): The starting point to search for data. If begin is None (default), use the most recent observed datetime (AKA sync_time).

    E.g. begin = 02:00

Search this region.           Ignore this, even if there's data.
/  /  /  /  /  /  /  /  /  |
-----|----------|----------|----------|----------|----------|
00:00      01:00      02:00      03:00      04:00      05:00

  • params (Optional[Dict[str, Any]], default None): The standard Meerschaum params query dictionary.

  • limit (Optional[int], default None): If provided, cap the number of rows to be returned.
  • fresh (bool, default False): If True, Ignore local cache and pull directly from the instance connector. Only comes into effect if a pipe was created with cache=True.
  • debug (bool default False): Verbosity toggle.
  • **kw (Any): Also end (exclusive), select_columns, omit_columns, order (default 'desc'), as_docs, as_polars, as_pandas and add_missing_columns, as in get_data(). Other keywords go to the instance connector; a keyword nothing reads is ignored with a warning that suggests the parameter you probably meant.
Returns

  • **A DataFrame (Polars or pandas, per meerschaum** (dataframe, as get_data()) of the pipe's):

  • data corresponding to the provided parameters. Backtrack data is a convenient way to get a

  • pipe's data "backtracked" from the most recent datetime.
def get_rowcount( self, begin: datetime.datetime | int | None = None, end: datetime.datetime | int | None = None, params: Dict[str, Any] | None = None, remote: bool = False, estimate: bool = False, silent: bool = False, debug: bool = False) -> int: 
365def get_rowcount(
366    self,
367    begin: Union[datetime, int, None] = None,
368    end: Union[datetime, int, None] = None,
369    params: Optional[Dict[str, Any]] = None,
370    remote: bool = False,
371    estimate: bool = False,
372    silent: bool = False,
373    debug: bool = False
374) -> int:
375    """
376    Get a Pipe's instance or remote rowcount.
377
378    Parameters
379    ----------
380    begin: Optional[datetime], default None
381        Count rows where datetime >= begin (inclusive).
382
383    end: Optional[datetime], default None
384        Count rows where datetime < end (exclusive).
385
386    remote: bool, default False
387        Count rows from a pipe's remote source.
388        **NOTE**: This is experimental!
389
390    estimate: bool, default False
391        If `True`, return the database's estimate of the number of rows from its catalog
392        statistics (PostgreSQL, TimescaleDB, MySQL / MariaDB, MSSQL, Oracle) instead of
393        counting them: instant on large tables, and close (the statistics trail recent
394        writes). The count is exact where the database has no estimate (SQLite, DuckDB), for
395        small tables, and with `begin`, `end`, `params` or `remote`.
396
397    silent: bool, default False
398        If `True`, return 0 instead of raising when the instance cannot be read
399        (Meerschaum 4's behavior).
400
401    debug: bool, default False
402        Verbosity toggle.
403
404    Returns
405    -------
406    An `int` of the number of rows in the pipe corresponding to the provided parameters.
407    Returned 0 if the pipe does not exist. Raises the database error if the instance cannot
408    be read (0 with `silent=True`).
409    """
410    from meerschaum import _core
411    try:
412        context = _rust_context(self, debug=debug)
413        begin, end = self.parse_date_bounds(begin, end, debug=debug)
414        return _core.pipe_get_rowcount(
415            self,
416            context,
417            begin=begin,
418            end=end,
419            params=params,
420            remote=remote,
421            estimate=estimate,
422            debug=debug,
423        )
424    except Exception:
425        if silent:
426            return 0
427        raise

Get a Pipe's instance or remote rowcount.

Parameters
  • begin (Optional[datetime], default None): Count rows where datetime >= begin (inclusive).
  • end (Optional[datetime], default None): Count rows where datetime < end (exclusive).
  • remote (bool, default False): Count rows from a pipe's remote source. NOTE: This is experimental!
  • estimate (bool, default False): If True, return the database's estimate of the number of rows from its catalog statistics (PostgreSQL, TimescaleDB, MySQL / MariaDB, MSSQL, Oracle) instead of counting them: instant on large tables, and close (the statistics trail recent writes). The count is exact where the database has no estimate (SQLite, DuckDB), for small tables, and with begin, end, params or remote.
  • silent (bool, default False): If True, return 0 instead of raising when the instance cannot be read (Meerschaum 4's behavior).
  • debug (bool, default False): Verbosity toggle.
Returns
  • An int of the number of rows in the pipe corresponding to the provided parameters.
  • Returned 0 if the pipe does not exist. Raises the database error if the instance cannot
  • be read (0 with silent=True).
def get_size(self, debug: bool = False, **kw: Any) -> int | None: 
430def get_size(
431    self,
432    debug: bool = False,
433    **kw: Any
434) -> Union[int, None]:
435    """
436    Return the on-disk size of the pipe's target table in bytes.
437
438    Parameters
439    ----------
440    debug: bool, default False
441        Verbosity toggle.
442
443    Returns
444    -------
445    An `int` of the number of bytes occupied by the pipe's target table,
446    or `None` if the size could not be determined (e.g. the connector does
447    not implement `get_pipe_size()` or the table does not exist).
448    """
449    from meerschaum.utils.warnings import warn
450    from meerschaum.utils.venv import Venv
451    from meerschaum.connectors import get_connector_plugin
452    from meerschaum.utils.misc import filter_keywords
453
454    connector = self.instance_connector
455    try:
456        with Venv(get_connector_plugin(connector)):
457            if not hasattr(connector, 'get_pipe_size'):
458                return None
459            kwargs = filter_keywords(
460                connector.get_pipe_size,
461                debug=debug,
462                **kw
463            )
464            return connector.get_pipe_size(self, **kwargs)
465    except NotImplementedError:
466        return None
467    except Exception as e:
468        warn(f"Failed to get the size of {self}:\n{e}", stack=False)
469    return None

Return the on-disk size of the pipe's target table in bytes.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
  • An int of the number of bytes occupied by the pipe's target table,
  • or None if the size could not be determined (e.g. the connector does
  • not implement get_pipe_size() or the table does not exist).
def get_doc(self, **kwargs) -> Dict[str, Any] | None: 
695def get_doc(self, **kwargs) -> Union[Dict[str, Any], None]:
696    """
697    Convenience function to return a single row as a dictionary (or `None`) from `Pipe.get_data().
698    Keywords arguments are passed to `Pipe.get_data()`.
699    """
700    from meerschaum.utils.warnings import warn
701    kwargs['limit'] = 1
702    kwargs['as_docs'] = True
703    try:
704        docs = self.get_data(**kwargs)
705        if not docs:
706            return None
707        return docs[0]
708    except Exception as e:
709        warn(f"Failed to read value from {self}:\n{e}", stack=False)
710        return None

Convenience function to return a single row as a dictionary (or None) from Pipe.get_data(). Keywords arguments are passed toPipe.get_data()`.

def get_docs(self, **kwargs) -> list[dict[str, typing.Any]]: 
712def get_docs(self, **kwargs) -> list[dict[str, Any]]:
713    """
714    Convenience method to return a pipe's data as a list of dictionaries.
715    Relies on `get_pipe_docs` from the instance connector if implemented.
716    """
717    kwargs['as_docs'] = True
718    return self.get_data(**kwargs)

Convenience method to return a pipe's data as a list of dictionaries. Relies on get_pipe_docs from the instance connector if implemented.

def get_value( self, column: str, params: Dict[str, Any] | None = None, **kwargs: Any) -> Any: 
720def get_value(
721    self,
722    column: str,
723    params: Optional[Dict[str, Any]] = None,
724    **kwargs: Any
725) -> Any:
726    """
727    Convenience function to return a single value (or `None`) from `Pipe.get_data()`.
728    Keywords arguments are passed to `Pipe.get_data()`.
729    """
730    from meerschaum.utils.warnings import warn
731    kwargs['select_columns'] = [column]
732    kwargs['limit'] = 1
733    kwargs['as_docs'] = True
734    try:
735        docs = self.get_data(params=params, **kwargs)
736        if not docs:
737            return None
738        if column not in docs[0]:
739            raise ValueError(f"Column '{column}' was not included in the result set.")
740        return docs[0][column]
741    except Exception as e:
742        warn(f"Failed to read value from {self}:\n{e}", stack=False)
743        return None

Convenience function to return a single value (or None) from Pipe.get_data(). Keywords arguments are passed to Pipe.get_data().

def get_chunk_interval( self, chunk_interval: datetime.timedelta | int | None = None, debug: bool = False) -> datetime.timedelta | int: 
472def get_chunk_interval(
473    self,
474    chunk_interval: Union[timedelta, int, None] = None,
475    debug: bool = False,
476) -> Union[timedelta, int]:
477    """
478    Get the chunk interval to use for this pipe.
479
480    The size is read from the `verify` parameters. Any one of these aliased keys may be used
481    (the first present, in this priority order, wins):
482
483        - `verify.chunk_minutes` (the default; 43200 — 30 days — if none is set)
484        - `verify.chunk_hours`
485        - `verify.chunk_days`
486        - `verify.chunk_weeks`
487        - `verify.chunk_years`
488        - `verify.chunk_seconds`
489
490    For an integer datetime axis, `verify.chunk_range` (if set) is used verbatim as the chunk size
491    in epoch units. Otherwise the time-based size above is converted to epoch units via the pipe's
492    `precision`, or — preserving legacy behavior when no `precision` is set — its minutes are used
493    verbatim.
494
495    Parameters
496    ----------
497    chunk_interval: Union[timedelta, int, None], default None
498        If provided, use this value instead of the `verify` keys.
499        An `int` is already in the axis's own units — epoch units for an integer axis,
500        minutes otherwise — and is used verbatim. A `timedelta` is a duration, so on an
501        integer axis it is scaled into epoch units by the pipe's `precision`.
502
503    Returns
504    -------
505    The chunk interval (`timedelta` or `int`) to use with this pipe's `datetime` axis.
506    """
507    from meerschaum import _core
508    return _core.get_chunk_interval(self, chunk_interval, self.dtypes)

Get the chunk interval to use for this pipe.

The size is read from the verify parameters. Any one of these aliased keys may be used (the first present, in this priority order, wins):

- `verify.chunk_minutes` (the default; 43200 — 30 days — if none is set)
- `verify.chunk_hours`
- `verify.chunk_days`
- `verify.chunk_weeks`
- `verify.chunk_years`
- `verify.chunk_seconds`

For an integer datetime axis, verify.chunk_range (if set) is used verbatim as the chunk size in epoch units. Otherwise the time-based size above is converted to epoch units via the pipe's precision, or, preserving legacy behavior when no precision is set, its minutes are used verbatim.

Parameters
  • chunk_interval (Union[timedelta, int, None], default None): If provided, use this value instead of the verify keys. An int is already in the axis's own units, epoch units for an integer axis, minutes otherwise, and is used verbatim. A timedelta is a duration, so on an integer axis it is scaled into epoch units by the pipe's precision.
Returns
  • The chunk interval (timedelta or int) to use with this pipe's datetime axis.
def get_chunk_bounds( self, begin: datetime.datetime | int | None = None, end: datetime.datetime | int | None = None, bounded: bool = False, chunk_interval: datetime.timedelta | int | None = None, align: bool = False, debug: bool = False) -> List[Tuple[datetime.datetime | int | None, datetime.datetime | int | None]]: 
511def get_chunk_bounds(
512    self,
513    begin: Union[datetime, int, None] = None,
514    end: Union[datetime, int, None] = None,
515    bounded: bool = False,
516    chunk_interval: Union[timedelta, int, None] = None,
517    align: bool = False,
518    debug: bool = False,
519) -> List[
520    Tuple[
521        Union[datetime, int, None],
522        Union[datetime, int, None],
523    ]
524]:
525    """
526    Return a list of datetime bounds for iterating over the pipe's `datetime` axis.
527
528    Parameters
529    ----------
530    begin: Union[datetime, int, None], default None
531        If provided, do not select less than this value.
532        Otherwise the first chunk will be unbounded.
533
534    end: Union[datetime, int, None], default None
535        If provided, do not select greater than or equal to this value.
536        Otherwise the last chunk will be unbounded.
537
538    bounded: bool, default False
539        If `True`, do not include `None` in the first chunk.
540
541    chunk_interval: Union[timedelta, int, None], default None
542        If provided, use this interval for the size of chunk boundaries.
543        The default value for this pipe may be set
544        under `pipe.parameters['verify']['chunk_minutes']`.
545
546    align: bool, default False
547        If `True`, anchor the interior chunk boundaries to a fixed Unix-epoch grid (the same
548        grid used for native range partitioning) rather than to `begin`. This makes the
549        boundaries deterministic across re-syncs and aligned with the pipe's partitions
550        (used by `Pipe.verify()`). The first chunk's lower bound and the last chunk's upper
551        bound are still clamped to `begin` / `end`.
552
553    debug: bool, default False
554        Verbosity toggle.
555
556    Returns
557    -------
558    A list of chunk bounds (datetimes or integers).
559    If unbounded, the first and last chunks will include `None`.
560    """
561    from meerschaum import _core
562    ### The sync times stand in for an unset `begin` / `end` (v4 queries them the same way).
563    oldest = self.get_sync_time(newest=False, debug=debug) if begin is None else None
564    newest = self.get_sync_time(newest=True, debug=debug) if end is None else None
565    return _core.get_chunk_bounds(
566        self.parse_date_bounds(begin, debug=debug),
567        self.parse_date_bounds(end, debug=debug),
568        bounded,
569        self.get_chunk_interval(chunk_interval, debug=debug),
570        align,
571        self.parse_date_bounds(oldest, debug=debug),
572        self.parse_date_bounds(newest, debug=debug),
573    )

Return a list of datetime bounds for iterating over the pipe's datetime axis.

Parameters
  • begin (Union[datetime, int, None], default None): If provided, do not select less than this value. Otherwise the first chunk will be unbounded.
  • end (Union[datetime, int, None], default None): If provided, do not select greater than or equal to this value. Otherwise the last chunk will be unbounded.
  • bounded (bool, default False): If True, do not include None in the first chunk.
  • chunk_interval (Union[timedelta, int, None], default None): If provided, use this interval for the size of chunk boundaries. The default value for this pipe may be set under pipe.parameters['verify']['chunk_minutes'].
  • align (bool, default False): If True, anchor the interior chunk boundaries to a fixed Unix-epoch grid (the same grid used for native range partitioning) rather than to begin. This makes the boundaries deterministic across re-syncs and aligned with the pipe's partitions (used by Pipe.verify()). The first chunk's lower bound and the last chunk's upper bound are still clamped to begin / end.
  • debug (bool, default False): Verbosity toggle.
Returns
  • A list of chunk bounds (datetimes or integers).
  • If unbounded, the first and last chunks will include None.
def get_chunk_bounds_batches( self, chunk_bounds: List[Tuple[datetime.datetime | int | None, datetime.datetime | int | None]], batchsize: int | None = None, workers: int | None = None, debug: bool = False) -> List[Tuple[Tuple[datetime.datetime | int | None, datetime.datetime | int | None], ...]]: 
576def get_chunk_bounds_batches(
577    self,
578    chunk_bounds: List[Tuple[Union[datetime, int, None], Union[datetime, int, None]]],
579    batchsize: Optional[int] = None,
580    workers: Optional[int] = None,
581    debug: bool = False,
582) -> List[
583    Tuple[
584        Tuple[
585            Union[datetime, int, None],
586            Union[datetime, int, None],
587        ], ...
588    ]
589]:
590    """
591    Return a list of tuples of chunk bounds of size `batchsize`.
592
593    Parameters
594    ----------
595    chunk_bounds: List[Tuple[Union[datetime, int, None], Union[datetime, int, None]]]
596        A list of chunk_bounds (see `Pipe.get_chunk_bounds()`).
597
598    batchsize: Optional[int], default None
599        How many chunks to include in a batch. Defaults to `Pipe.get_num_workers()`.
600
601    workers: Optional[int], default None
602        If `batchsize` is `None`, use this as the desired number of workers.
603        Passed to `Pipe.get_num_workers()`.
604
605    Returns
606    -------
607    A list of tuples of chunk bound tuples.
608    """
609    from meerschaum.utils.misc import iterate_chunks
610    
611    if batchsize is None:
612        batchsize = self.get_num_workers(workers=workers)
613
614    return [
615        tuple(
616            _batch_chunk_bounds
617            for _batch_chunk_bounds in batch
618            if _batch_chunk_bounds is not None
619        )
620        for batch in iterate_chunks(chunk_bounds, batchsize)
621        if batch
622    ]

Return a list of tuples of chunk bounds of size batchsize.

Parameters
  • chunk_bounds (List[Tuple[Union[datetime, int, None], Union[datetime, int, None]]]): A list of chunk_bounds (see Pipe.get_chunk_bounds()).
  • batchsize (Optional[int], default None): How many chunks to include in a batch. Defaults to Pipe.get_num_workers().
  • workers (Optional[int], default None): If batchsize is None, use this as the desired number of workers. Passed to Pipe.get_num_workers().
Returns
  • A list of tuples of chunk bound tuples.
def parse_date_bounds( self, *dt_vals: datetime.datetime | int | None, debug: bool = False) -> datetime.datetime | int | str | None | Tuple[datetime.datetime | int | str | None]: 
625def parse_date_bounds(self, *dt_vals: Union[datetime, int, None], debug: bool = False) -> Union[
626    datetime,
627    int,
628    str,
629    None,
630    Tuple[Union[datetime, int, str, None]]
631]:
632    """
633    Given a date bound (begin, end), coerce a timezone if necessary.
634    """
635    from meerschaum.utils.misc import is_int
636    from meerschaum.utils.dtypes import coerce_timezone, MRSM_PD_DTYPES, are_dtypes_equal
637    from meerschaum.utils.warnings import warn
638
639    _columns = None
640    _dtypes = None
641
642    def _get_coercion_info():
643        nonlocal _columns, _dtypes
644        if _columns is None:
645            _columns = self.get_parameters(debug=debug).get('columns', {}) or {}
646        if _dtypes is None:
647            _dtypes = self.get_dtypes(debug=debug)
648
649    def _parse_date_bound(dt_val):
650        if dt_val is None:
651            return None
652
653        if isinstance(dt_val, int):
654            return dt_val
655
656        if dt_val == '':
657            return ''
658
659        if is_int(dt_val):
660            return int(dt_val)
661
662        if isinstance(dt_val, str):
663            ### Only a text bound needs `dateutil` (a missing package is not installed
664            ### for the integer, `None` and datetime bounds that never reach here).
665            dateutil_parser = mrsm.attempt_import('dateutil.parser')
666            try:
667                dt_val = dateutil_parser.parse(dt_val)
668            except Exception as e:
669                warn(f"Could not parse '{dt_val}' as datetime:\n{e}")
670                return None
671
672        _get_coercion_info()
673        dt_col = _columns.get('datetime', None)
674        dt_typ = str(_dtypes.get(dt_col, 'datetime'))
675        if are_dtypes_equal(dt_typ, 'int'):
676            if self.get_parameters(debug=debug).get('precision'):
677                from meerschaum.utils.dtypes import datetime_to_int
678                return datetime_to_int(dt_val, self.precision['unit'])
679            from meerschaum.utils.warnings import error
680            error(
681                f"Cannot use datetime bound '{dt_val}' on the non-epoch integer axis "
682                f"of {self}.\n    Pass an integer instead, or set the `precision` parameter.",
683                ValueError,
684            )
685        if dt_typ == 'datetime':
686            dt_typ = MRSM_PD_DTYPES['datetime']
687        return coerce_timezone(dt_val, strip_utc=('utc' not in dt_typ.lower()))
688
689    bounds = tuple(_parse_date_bound(dt_val) for dt_val in dt_vals)
690    if len(bounds) == 1:
691        return bounds[0]
692    return bounds

Given a date bound (begin, end), coerce a timezone if necessary.

def register(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
12def register(
13    self,
14    debug: bool = False,
15    **kw: Any
16) -> SuccessTuple:
17    """
18    Register a new Pipe along with its attributes.
19
20    Parameters
21    ----------
22    debug: bool, default False
23        Verbosity toggle.
24
25    kw: Any
26        Keyword arguments to pass to `instance_connector.register_pipe()`.
27
28    Returns
29    -------
30    A `SuccessTuple` of success, message.
31    """
32    if self.temporary:
33        return False, "Cannot register pipes created with `temporary=True` (read-only)."
34
35    from meerschaum.utils.formatting import print_exception
36    from meerschaum.utils.venv import Venv
37    from meerschaum.connectors import get_connector_plugin, custom_types
38    from meerschaum.config._patch import apply_patch_to_config
39
40    import warnings
41    with warnings.catch_warnings():
42        warnings.simplefilter('ignore')
43        try:
44            _conn = self.connector
45        except Exception:
46            _conn = None
47
48        if isinstance(_conn, str):
49            _conn = None
50
51    if (
52        _conn is not None
53        and
54        (_conn.type == 'plugin' or _conn.type in custom_types)
55        and
56        getattr(_conn, 'register', None) is not None
57    ):
58        try:
59            with Venv(get_connector_plugin(_conn), debug=debug):
60                params = self.connector.register(self)
61        except Exception:
62            print_exception()
63            params = None
64        params = {} if params is None else params
65        if not isinstance(params, dict):
66            from meerschaum.utils.warnings import warn
67            warn(
68                f"Invalid parameters returned from `register()` in connector {self.connector}:\n"
69                + f"{params}"
70            )
71        else:
72            self.parameters = apply_patch_to_config(params, self.parameters)
73
74    if not self.parameters:
75        cols = self.columns if self.columns else {'datetime': None, 'id': None}
76        self.parameters = {
77            'columns': cols,
78        }
79
80    ### Refuse an index column which cannot be sorted or deduplicated, rather than
81    ### registering a pipe whose second sync is guaranteed to fail.
82    indices_success, indices_msg = self.validate_indices(debug=debug)
83    if not indices_success:
84        return indices_success, indices_msg
85
86    ### Registering a pipe that exists again must not get past its `parameters` lock (a custom
87    ### instance connector may overwrite the registration; the core checks its own instances).
88    refusal = self._edit_lock_refusal(patch=False, debug=debug)
89    if refusal:
90        return False, refusal
91
92    with Venv(get_connector_plugin(self.instance_connector)):
93        result = self.instance_connector.register_pipe(self, debug=debug, **kw)
94    from meerschaum.core.Pipe._cache import _bump_invalidation_epoch
95    _bump_invalidation_epoch()
96    return result

Register a new Pipe along with its attributes.

Parameters
  • debug (bool, default False): Verbosity toggle.
  • kw (Any): Keyword arguments to pass to instance_connector.register_pipe().
Returns
attributes: Dict[str, Any] 
23@property
24def attributes(self) -> Dict[str, Any]:
25    """
26    Return a dictionary of a pipe's keys and parameters.
27    These values are reflected directly from the pipes table of the instance.
28    """
29    from meerschaum.config import get_config
30    from meerschaum.config._patch import apply_patch_to_config
31    from meerschaum.utils.venv import Venv
32    from meerschaum.connectors import get_connector_plugin
33    from meerschaum.utils.dtypes import get_current_timestamp
34
35    if self.temporary:
36        return self._attributes
37
38    with self._attributes_lock:
39        # Instance connectors may read local parameters while fetching metadata.
40        if self.__dict__.get('_attributes_loading', False):
41            return self._attributes
42        timeout_seconds = get_config('pipes', 'attributes', 'local_cache_timeout_seconds')
43        now = get_current_timestamp('ms', as_int=True) / 1000
44        sync_time = self._get_cached_value('_attributes_sync_time', debug=self.debug)
45        timed_out = (
46            sync_time is None
47            or (timeout_seconds is not None and (now - sync_time) >= timeout_seconds)
48        )
49        if timed_out:
50            local_attributes = self._get_cached_value('attributes', debug=self.debug) or {}
51            self._attributes_loading = True
52            try:
53                with Venv(get_connector_plugin(self.instance_connector)):
54                    instance_attributes = self.instance_connector.get_pipe_attributes(self)
55                if not isinstance(instance_attributes, dict) or (
56                    not instance_attributes and self._get_cached_value('_id') is not None
57                ):
58                    raise RuntimeError(f"Missing metadata for registered pipe {self}.")
59                self._cache_value(
60                    'attributes',
61                    apply_patch_to_config(instance_attributes, local_attributes),
62                    memory_only=True,
63                    debug=self.debug,
64                )
65            except Exception as e:
66                self._attributes_load_failures = self.__dict__.get('_attributes_load_failures', 0) + 1
67                self._clear_cache_key('_attributes_sync_time', debug=self.debug)
68                warn(f"Failed to load attributes for {self}: {e}")
69                return self._attributes
70            finally:
71                self._attributes_loading = False
72            self._cache_value(
73                '_attributes_sync_time', get_current_timestamp('ms', as_int=True) / 1000,
74                memory_only=True, debug=self.debug,
75            )
76
77        return self._attributes

Return a dictionary of a pipe's keys and parameters. These values are reflected directly from the pipes table of the instance.

parameters: Dict[str, Any] | None 
214@property
215def parameters(self) -> Optional[Dict[str, Any]]:
216    """
217    Return the parameters dictionary of the pipe.
218
219    The dictionary is a live view (design §4.4): reads are resolved, and mutations (including
220    nested ones) write through to the raw parameters, so
221    `pipe.parameters['fetch']['backtrack_minutes'] = 5` followed by `pipe.edit()` persists.
222    """
223    resolved = self.get_parameters(debug=self.debug)
224    from ._live_parameters import live_view
225    return live_view(self, resolved)

Return the parameters dictionary of the pipe.

The dictionary is a live view (design §4.4): reads are resolved, and mutations (including nested ones) write through to the raw parameters, so pipe.parameters['fetch']['backtrack_minutes'] = 5 followed by pipe.edit() persists.

columns: Dict[str, str] | None 
238@property
239def columns(self) -> Union[Dict[str, str], None]:
240    """
241    Return the `columns` dictionary defined in `meerschaum.Pipe.parameters`.
242    """
243    cols = self.get_parameters(debug=self.debug).get('columns', {})
244    if not isinstance(cols, dict):
245        return {}
246    return {col_ix: col for col_ix, col in cols.items() if col and col_ix}

Return the columns dictionary defined in meerschaum.Pipe.parameters.

indices: Dict[str, str | List[str]] | None 
263@property
264def indices(self) -> Union[Dict[str, Union[str, List[str]]], None]:
265    """
266    Return the `indices` dictionary defined in `meerschaum.Pipe.parameters`.
267    """
268    _parameters = self.get_parameters(debug=self.debug)
269    indices_key = (
270        'indexes'
271        if 'indexes' in _parameters
272        else 'indices'
273    )
274
275    _indices = _parameters.get(indices_key, {})
276    _columns = self.columns
277    dt_col = _columns.get('datetime', None)
278    if not isinstance(_indices, dict):
279        _indices = {}
280    unique_cols = list(set((
281        [dt_col]
282        if dt_col
283        else []
284    ) + [
285        col
286        for col_ix, col in _columns.items()
287        if col and col_ix != 'datetime'
288    ]))
289    return {
290        **({'unique': unique_cols} if len(unique_cols) > 1 else {}),
291        **{col_ix: col for col_ix, col in _columns.items() if col},
292        **_indices
293    }

Return the indices dictionary defined in meerschaum.Pipe.parameters.

indexes: Dict[str, str | List[str]] | None 
296@property
297def indexes(self) -> Union[Dict[str, Union[str, List[str]]], None]:
298    """
299    Alias for `meerschaum.Pipe.indices`.
300    """
301    return self.indices

Alias for meerschaum.Pipe.indices.

dtypes: Dict[str, Any] 
352@property
353def dtypes(self) -> Dict[str, Any]:
354    """
355    If defined, return the `dtypes` dictionary defined in `meerschaum.Pipe.parameters`.
356    """
357    return self.get_dtypes(refresh=False, debug=self.debug)

If defined, return the dtypes dictionary defined in meerschaum.Pipe.parameters.

autoincrement: bool 
510@property
511def autoincrement(self) -> bool:
512    """
513    Return the `autoincrement` parameter for the pipe.
514    """
515    return self.get_parameters(debug=self.debug).get('autoincrement', False)

Return the autoincrement parameter for the pipe.

autotime: bool 
526@property
527def autotime(self) -> bool:
528    """
529    Return the `autotime` parameter for the pipe.
530    """
531    return self.get_parameters(debug=self.debug).get('autotime', False)

Return the autotime parameter for the pipe.

upsert: bool 
477@property
478def upsert(self) -> bool:
479    """
480    Return whether `upsert` is set for the pipe.
481    """
482    return self.get_parameters(debug=self.debug).get('upsert', False)

Return whether upsert is set for the pipe.

static: bool 
493@property
494def static(self) -> bool:
495    """
496    Return whether `static` is set for the pipe.
497    """
498    return self.get_parameters(debug=self.debug).get('static', False)

Return whether static is set for the pipe.

tzinfo: None | datetime.timezone 
542@property
543def tzinfo(self) -> Union[None, timezone]:
544    """
545    Return `timezone.utc` if the pipe is timezone-aware.
546    """
547    _tzinfo = self._get_cached_value('tzinfo', debug=self.debug)
548    if _tzinfo is not None:
549        return _tzinfo if _tzinfo != 'None' else None
550
551    _tzinfo = None
552    dt_col = self.columns.get('datetime', None)
553    dt_typ = str(self.dtypes.get(dt_col, 'datetime')) if dt_col else None
554    if self.autotime:
555        ts_col = mrsm.get_config('pipes', 'autotime', 'column_name_if_datetime_missing')
556        ts_typ = self.dtypes.get(ts_col, 'datetime')
557        dt_typ = ts_typ
558
559    if dt_typ and 'utc' in dt_typ.lower() or dt_typ == 'datetime':
560        _tzinfo = timezone.utc
561
562    self._cache_value('tzinfo', (_tzinfo if _tzinfo is not None else 'None'), debug=self.debug)
563    return _tzinfo

Return timezone.utc if the pipe is timezone-aware.

enforce: bool 
566@property
567def enforce(self) -> bool:
568    """
569    Return the `enforce` parameter for the pipe.
570    """
571    return self.get_parameters(debug=self.debug).get('enforce', True)

Return the enforce parameter for the pipe.

null_indices: bool 
582@property
583def null_indices(self) -> bool:
584    """
585    Return the `null_indices` parameter for the pipe.
586    """
587    return self.get_parameters(debug=self.debug).get('null_indices', True)

Return the null_indices parameter for the pipe.

mixed_numerics: bool 
598@property
599def mixed_numerics(self) -> bool:
600    """
601    Return the `mixed_numerics` parameter for the pipe.
602    """
603    return self.get_parameters(debug=self.debug).get('mixed_numerics', True)

Return the mixed_numerics parameter for the pipe.

def get_columns(self, *args: str, error: bool = False) -> str | Tuple[str]: 
614def get_columns(self, *args: str, error: bool = False) -> Union[str, Tuple[str]]:
615    """
616    Check if the requested columns are defined.
617
618    Parameters
619    ----------
620    *args: str
621        The column names to be retrieved.
622
623    error: bool, default False
624        If `True`, raise an `Exception` if the specified column is not defined.
625
626    Returns
627    -------
628    A tuple of the same size of `args` or a `str` if `args` is a single argument.
629
630    Examples
631    --------
632    >>> pipe = mrsm.Pipe('test', 'test')
633    >>> pipe.columns = {'datetime': 'dt', 'id': 'id'}
634    >>> pipe.get_columns('datetime', 'id')
635    ('dt', 'id')
636    >>> pipe.get_columns('value', error=True)
637    Exception:  🛑 Missing 'value' column for Pipe('test', 'test').
638    """
639    from meerschaum.utils.warnings import error as _error
640    if not args:
641        args = tuple(self.columns.keys())
642    col_names = []
643    for col in args:
644        col_name = None
645        try:
646            col_name = self.columns[col]
647            if col_name is None and error:
648                _error(f"Please define the name of the '{col}' column for {self}.")
649        except Exception:
650            col_name = None
651        if col_name is None and error:
652            _error(f"Missing '{col}'" + f" column for {self}.")
653        col_names.append(col_name)
654    if len(col_names) == 1:
655        return col_names[0]
656    return tuple(col_names)

Check if the requested columns are defined.

Parameters
  • *args (str): The column names to be retrieved.
  • error (bool, default False): If True, raise an Exception if the specified column is not defined.
Returns
  • A tuple of the same size of args or a str if args is a single argument.
Examples
>>> pipe = mrsm.Pipe('test', 'test')
>>> pipe.columns = {'datetime': 'dt', 'id': 'id'}
>>> pipe.get_columns('datetime', 'id')
('dt', 'id')
>>> pipe.get_columns('value', error=True)
Exception:  🛑 Missing 'value' column for Pipe('test', 'test').
def get_columns_types( self, refresh: bool = False, debug: bool = False) -> Dict[str, str] | None: 
659def get_columns_types(
660    self,
661    refresh: bool = False,
662    debug: bool = False,
663) -> Union[Dict[str, str], None]:
664    """
665    Get a dictionary of a pipe's column names and their types.
666
667    Parameters
668    ----------
669    refresh: bool, default False
670        If `True`, invalidate the cache and fetch directly from the instance connector.
671
672    debug: bool, default False:
673        Verbosity toggle.
674
675    Returns
676    -------
677    A dictionary of column names (`str`) to column types (`str`).
678
679    Examples
680    --------
681    >>> pipe.get_columns_types()
682    {
683      'dt': 'TIMESTAMP WITH TIMEZONE',
684      'id': 'BIGINT',
685      'val': 'DOUBLE PRECISION',
686    }
687    >>>
688    """
689    from meerschaum.connectors import get_connector_plugin
690    from meerschaum.utils.dtypes import get_current_timestamp
691
692    now = get_current_timestamp('ms', as_int=True) / 1000
693    cache_seconds = (
694        mrsm.get_config('pipes', 'static', 'static_schema_cache_seconds')
695        if self.static
696        else mrsm.get_config('pipes', 'dtypes', 'columns_types_cache_seconds')
697    )
698    if refresh:
699        self._clear_cache_key('_columns_types_timestamp', debug=debug)
700        self._clear_cache_key('_columns_types', debug=debug)
701
702    _columns_types = self._get_cached_value('_columns_types', debug=debug)
703    if _columns_types:
704        columns_types_timestamp = self._get_cached_value('_columns_types_timestamp', debug=debug)
705        if columns_types_timestamp is not None:
706            delta = now - columns_types_timestamp
707            if delta < cache_seconds:
708                if debug:
709                    dprint(
710                        f"Returning cached `columns_types` for {self} "
711                        f"({round(delta, 2)} seconds old)."
712                    )
713                return _columns_types
714
715    with mrsm.Venv(get_connector_plugin(self.instance_connector)):
716        _columns_types = (
717            self.instance_connector.get_pipe_columns_types(self, debug=debug)
718            if hasattr(self.instance_connector, 'get_pipe_columns_types')
719            else None
720        )
721
722    self._cache_value('_columns_types', _columns_types, debug=debug)
723    self._cache_value('_columns_types_timestamp', now, debug=debug)
724    return _columns_types or {}

Get a dictionary of a pipe's column names and their types.

Parameters
  • refresh (bool, default False): If True, invalidate the cache and fetch directly from the instance connector.
  • debug (bool, default False:): Verbosity toggle.
Returns
  • A dictionary of column names (str) to column types (str).
Examples
>>> pipe.get_columns_types()
{
  'dt': 'TIMESTAMP WITH TIMEZONE',
  'id': 'BIGINT',
  'val': 'DOUBLE PRECISION',
}
>>>
def get_columns_indices( self, debug: bool = False, refresh: bool = False) -> Dict[str, List[Dict[str, str]]]: 
727def get_columns_indices(
728    self,
729    debug: bool = False,
730    refresh: bool = False,
731) -> Dict[str, List[Dict[str, str]]]:
732    """
733    Return a dictionary mapping columns to index information.
734    """
735    from meerschaum.connectors import get_connector_plugin
736    from meerschaum.utils.dtypes import get_current_timestamp
737
738    now = get_current_timestamp('ms', as_int=True) / 1000
739    cache_seconds = (
740        mrsm.get_config('pipes', 'static', 'static_schema_cache_seconds')
741        if self.static
742        else mrsm.get_config('pipes', 'dtypes', 'columns_types_cache_seconds')
743    )
744    if refresh:
745        self._clear_cache_key('_columns_indices_timestamp', debug=debug)
746        self._clear_cache_key('_columns_indices', debug=debug)
747
748    _columns_indices = self._get_cached_value('_columns_indices', debug=debug)
749
750    if _columns_indices:
751        columns_indices_timestamp = self._get_cached_value('_columns_indices_timestamp', debug=debug)
752        if columns_indices_timestamp is not None:
753            delta = now - columns_indices_timestamp
754            if delta < cache_seconds:
755                if debug:
756                    dprint(
757                        f"Returning cached `columns_indices` for {self} "
758                        f"({round(delta, 2)} seconds old)."
759                    )
760                return _columns_indices
761
762    with mrsm.Venv(get_connector_plugin(self.instance_connector)):
763        _columns_indices = (
764            self.instance_connector.get_pipe_columns_indices(self, debug=debug)
765            if hasattr(self.instance_connector, 'get_pipe_columns_indices')
766            else None
767        )
768
769    self._cache_value('_columns_indices', _columns_indices, debug=debug)
770    self._cache_value('_columns_indices_timestamp', now, debug=debug)
771    return {k: v for k, v in _columns_indices.items() if k and v} or {}

Return a dictionary mapping columns to index information.

def get_indices(self) -> Dict[str, str]: 
1175def get_indices(self) -> Dict[str, str]:
1176    """
1177    Return a dictionary mapping index keys to their names in the database.
1178
1179    Returns
1180    -------
1181    A dictionary of index keys to index names.
1182    """
1183    from meerschaum.connectors import get_connector_plugin
1184    with mrsm.Venv(get_connector_plugin(self.instance_connector)):
1185        if hasattr(self.instance_connector, 'get_pipe_index_names'):
1186            result = self.instance_connector.get_pipe_index_names(self)
1187        else:
1188            result = {}
1189    
1190    return result

Return a dictionary mapping index keys to their names in the database.

Returns
  • A dictionary of index keys to index names.
def get_parameters( self, apply_symlinks: bool = True, refresh: bool = False, debug: bool = False) -> Dict[str, Any]: 
109def get_parameters(
110    self,
111    apply_symlinks: bool = True,
112    refresh: bool = False,
113    debug: bool = False,
114) -> Dict[str, Any]:
115    """
116    Return the `parameters` dictionary of the pipe.
117
118    Parameters
119    ----------
120    apply_symlinks: bool, default True
121        If `True`, resolve references to parameters from other pipes.
122
123    refresh: bool, default False
124        If `True`, pull the latest attributes for the pipe.
125
126    Returns
127    -------
128    The pipe's parameters dictionary.
129    """
130    from copy import deepcopy
131    from meerschaum import _core
132
133    if refresh:
134        _ = self._invalidate_cache(hard=True)
135        with self._attributes_lock:
136            self._clear_cache_key('_attributes_sync_time', debug=debug)
137            self.__dict__.pop('_target', None)
138        ### Drop any memoized resolution so a later non-refresh call recomputes from fresh state.
139        _ = self.__dict__.pop('_resolved_parameters_raw', None)
140        _ = self.__dict__.pop('_resolved_parameters', None)
141        _ = self.__dict__.pop('_resolved_parameters_symlinks', None)
142
143    raw_parameters = self.attributes.get('parameters', {})
144    if not apply_symlinks:
145        return raw_parameters
146
147    # Resolving a self-table template asks the Python connector for this Pipe's target
148    # and schema. Those access parameters again; use the current raw snapshot for that
149    # nested metadata read rather than recursively re-entering the native resolver.
150    # Context-local keys also cover fresh wrappers for the same Pipe without affecting
151    # concurrent readers or caching an incompletely resolved result.
152    resolution_key = (self.instance_keys, self.connector_keys, self.metric_key, self.location_key)
153    resolution_stack = _PARAMETER_RESOLUTION_STACK.get()
154    if resolution_key in resolution_stack:
155        return deepcopy(raw_parameters)
156
157    ### Resolving references + `{{ Pipe() }}` / `MRSM{}` symlinks (in Rust) is expensive (it walks
158    ### reference pipes and may build connectors), and `get_parameters` is a hot path hit by
159    ### `.dtypes`, `.columns`, `.precision`, etc. Memoize the resolved result, keyed on the
160    ### mutation counter of the pipe's `_core.Pipe` (the Python view objects keep their identity
161    ### across mutations). Schema is *not* part of this: dynamic-schema freshness is handled
162    ### separately by `get_columns_types`' TTL cache, so this is safe for dynamic pipes.
163    can_memoize = not refresh
164    core = self._core_pipe
165    ### Cross-pipe `{{ Pipe() }}` templates read other pipes' state. v4 keeps them memoized
166    ### forever; here the memo is also keyed on this process's metadata writes (a hard
167    ### `_invalidate_cache()`, `register()`, or any `_core.SqlConnector` register / edit / delete,
168    ### including a sync's dtype edits) and expires after
169    ### `pipes:attributes:local_cache_timeout_seconds`, v4's own staleness window for attributes
170    ### (another process's edit of a referenced pipe shows up within it). Re-resolving on every
171    ### access cost one instance round trip per reference per access (wave 9 U).
172    cross_pipe = _has_cross_pipe_template(self, core, raw_parameters)
173    if cross_pipe:
174        from meerschaum.core.Pipe._cache import _INVALIDATION_EPOCH
175        memo_key = (
176            core.generation,
177            _INVALIDATION_EPOCH[0],
178            _core.pipe_metadata_epoch(),
179        )
180    else:
181        memo_key = core.generation
182    memo_hit = (
183        '_resolved_parameters' in self.__dict__
184        and self.__dict__.get('_resolved_parameters_raw', None) == memo_key
185        and (not cross_pipe or _cross_pipe_memo_is_fresh(self))
186    )
187    if can_memoize and memo_hit:
188        self._symlinks = self.__dict__.get('_resolved_parameters_symlinks', {})
189        ### Return a copy so callers that mutate the result (e.g. `infer_dtypes(persist=True)`)
190        ### don't corrupt the memo.
191        return deepcopy(self.__dict__['_resolved_parameters'])
192
193    ### References, `{{ Pipe() }}` and `MRSM{}` (against the live config registry) resolve in
194    ### `meerschaum_core`; the Rust call also returns the `_symlinks` v4 records on the pipe.
195    resolution_token = _PARAMETER_RESOLUTION_STACK.set((*resolution_stack, resolution_key))
196    try:
197        resolved_parameters, self._symlinks = _core.get_parameters(self)
198    finally:
199        _PARAMETER_RESOLUTION_STACK.reset(resolution_token)
200
201    if can_memoize:
202        ### Stash the symlinks captured above alongside the resolved result.
203        if cross_pipe:
204            import time
205            self.__dict__['_resolved_parameters_ts'] = time.monotonic()
206        self.__dict__['_resolved_parameters_raw'] = memo_key
207        self.__dict__['_resolved_parameters'] = resolved_parameters
208        self.__dict__['_resolved_parameters_symlinks'] = self._symlinks
209        return deepcopy(resolved_parameters)
210
211    return resolved_parameters

Return the parameters dictionary of the pipe.

Parameters
  • apply_symlinks (bool, default True): If True, resolve references to parameters from other pipes.
  • refresh (bool, default False): If True, pull the latest attributes for the pipe.
Returns
  • The pipe's parameters dictionary.
def get_dtypes( self, infer: bool = True, refresh: bool = False, debug: bool = False) -> Dict[str, Any]: 
371def get_dtypes(
372    self,
373    infer: bool = True,
374    refresh: bool = False,
375    debug: bool = False,
376) -> Dict[str, Any]:
377    """
378    If defined, return the `dtypes` dictionary defined in `meerschaum.Pipe.parameters`.
379
380    Parameters
381    ----------
382    infer: bool, default True
383        If `True`, include the implicit existing dtypes.
384        Else only return the explicitly configured dtypes (e.g. `Pipe.parameters['dtypes']`).
385
386    refresh: bool, default False
387        If `True`, invalidate any cache and return the latest known dtypes.
388
389    Returns
390    -------
391    A dictionary mapping column names to dtypes.
392    """
393    from meerschaum.config._patch import apply_patch_to_config
394    from meerschaum.utils.dtypes import MRSM_ALIAS_DTYPES
395    parameters = self.get_parameters(refresh=refresh, debug=debug)
396    configured_dtypes = parameters.get('dtypes', {})
397    if debug:
398        dprint(f"Configured dtypes for {self}:")
399        mrsm.pprint(configured_dtypes)
400
401    remote_dtypes = (
402        self.infer_dtypes(persist=False, refresh=refresh, debug=debug)
403        if infer
404        else {}
405    )
406    if debug and infer:
407        dprint(f"Remote dtypes for {self}:")
408        mrsm.pprint(remote_dtypes)
409
410    patched_dtypes = apply_patch_to_config((remote_dtypes or {}), (configured_dtypes or {}))
411
412    dt_col = parameters.get('columns', {}).get('datetime', None)
413    primary_col = parameters.get('columns', {}).get('primary', None)
414    _dtypes = {
415        col: MRSM_ALIAS_DTYPES.get(typ, typ)
416        for col, typ in patched_dtypes.items()
417        if col and typ
418    }
419    if dt_col and dt_col not in configured_dtypes:
420        _dtypes[dt_col] = 'datetime'
421    if primary_col and parameters.get('autoincrement', False) and primary_col not in _dtypes:
422        _dtypes[primary_col] = 'int'
423
424    return _dtypes

If defined, return the dtypes dictionary defined in meerschaum.Pipe.parameters.

Parameters
  • infer (bool, default True): If True, include the implicit existing dtypes. Else only return the explicitly configured dtypes (e.g. Pipe.parameters['dtypes']).
  • refresh (bool, default False): If True, invalidate any cache and return the latest known dtypes.
Returns
  • A dictionary mapping column names to dtypes.
def update_parameters( self, parameters_patch: Dict[str, Any], persist: bool = True, debug: bool = False) -> Tuple[bool, str]: 
1193def update_parameters(
1194    self,
1195    parameters_patch: Dict[str, Any],
1196    persist: bool = True,
1197    debug: bool = False,
1198) -> mrsm.SuccessTuple:
1199    """
1200    Apply a patch to a pipe's `parameters` dictionary.
1201
1202    Parameters
1203    ----------
1204    parameters_patch: Dict[str, Any]
1205        The patch to be applied to `Pipe.parameters`.
1206
1207    persist: bool, default True
1208        If `True`, call `Pipe.edit()` to persist the new parameters.
1209    """
1210    from meerschaum.config import apply_patch_to_config
1211    if 'parameters' not in self._attributes:
1212        self._attributes['parameters'] = {}
1213
1214    self._attributes['parameters'] = apply_patch_to_config(
1215        self._attributes['parameters'],
1216        parameters_patch,
1217    )
1218
1219    if self.temporary:
1220        persist = False
1221
1222    if not persist:
1223        return True, "Success"
1224
1225    return self.edit(debug=debug)

Apply a patch to a pipe's parameters dictionary.

Parameters
  • parameters_patch (Dict[str, Any]): The patch to be applied to Pipe.parameters.
  • persist (bool, default True): If True, call Pipe.edit() to persist the new parameters.
def validate_indices(self, debug: bool = False) -> Tuple[bool, str]: 
431def validate_indices(self, debug: bool = False) -> mrsm.SuccessTuple:
432    """
433    Return whether this pipe's index columns can actually be used as indices.
434
435    A `json` column holds `dict` or `list` values. Syncing sorts and deduplicates
436    on the index columns in Pandas (`Series.unique()`, `DataFrame.sort_values()`),
437    and both raise `TypeError: unhashable type: 'dict'` on container values. The
438    failure surfaces on the *second* sync — the first one has nothing to diff
439    against — so it is worth reporting up front instead of at that point.
440
441    Note this is a client-side limitation, not a database one: on PostgreSQL the
442    `json` dtype is stored as `JSONB`, which does support B-tree, hash, and GIN
443    indices. Non-unique performance indices declared under `indices` are pure
444    DDL and are not affected; only the index *roles* in `columns` are.
445
446    Returns
447    -------
448    A `SuccessTuple` of whether the index columns are valid.
449    """
450    columns = self.columns or {}
451    if not columns:
452        return True, "Success"
453
454    dtypes = self.get_dtypes(infer=False, debug=debug)
455    offenders = {
456        col_ix: col
457        for col_ix, col in columns.items()
458        if col and str(dtypes.get(col, '')).lower() in UNHASHABLE_DTYPES
459    }
460    if not offenders:
461        return True, "Success"
462
463    listing = ', '.join(
464        f"'{col}' (as the '{col_ix}' index)"
465        for col_ix, col in sorted(offenders.items())
466    )
467    return False, (
468        f"Cannot use a 'json' column as an index column: {listing}.\n"
469        "    JSON values are dictionaries or lists, which cannot be sorted or "
470        "deduplicated when syncing.\n"
471        "    Either give the column a scalar dtype (e.g. 'str'), or move it out of "
472        "`columns` and\n"
473        "    declare it under `indices` if you only need a database index on it."
474    )

Return whether this pipe's index columns can actually be used as indices.

A json column holds dict or list values. Syncing sorts and deduplicates on the index columns in Pandas (Series.unique(), DataFrame.sort_values()), and both raise TypeError: unhashable type: 'dict' on container values. The failure surfaces on the second sync, the first one has nothing to diff against, so it is worth reporting up front instead of at that point.

Note this is a client-side limitation, not a database one: on PostgreSQL the json dtype is stored as JSONB, which does support B-tree, hash, and GIN indices. Non-unique performance indices declared under indices are pure DDL and are not affected; only the index roles in columns are.

Returns
tags: List[str] | None 
329@property
330def tags(self) -> Union[List[str], None]:
331    """
332    If defined, return the `tags` list defined in `meerschaum.Pipe.parameters`.
333    """
334    return self.parameters.get('tags', [])

If defined, return the tags list defined in meerschaum.Pipe.parameters.

def get_id(self, **kw: Any) -> int | str | None: 
774def get_id(self, **kw: Any) -> Union[int, str, None]:
775    """
776    Fetch a pipe's ID from its instance connector.
777    If the pipe is not registered, return `None`.
778    """
779    if self.temporary:
780        return None
781
782    from meerschaum.utils.venv import Venv
783    from meerschaum.connectors import get_connector_plugin
784
785    with Venv(get_connector_plugin(self.instance_connector)):
786        if hasattr(self.instance_connector, 'get_pipe_id'):
787            return self.instance_connector.get_pipe_id(self, **kw)
788
789    return None

Fetch a pipe's ID from its instance connector. If the pipe is not registered, return None.

id: int | str | uuid.UUID | None 
792@property
793def id(self) -> Union[int, str, uuid.UUID, None]:
794    """
795    Fetch and cache a pipe's ID.
796    """
797    _id = self._get_cached_value('_id', debug=self.debug)
798    if _id is None:
799        _id = self.get_id(debug=self.debug)
800        if _id is not None:
801            self._cache_value('_id', _id, debug=self.debug)
802    return _id

Fetch and cache a pipe's ID.

def get_val_column(self, debug: bool = False) -> str | None: 
805def get_val_column(self, debug: bool = False) -> Union[str, None]:
806    """
807    Return the name of the value column if it's defined, otherwise make an educated guess.
808    If not set in the `columns` dictionary, return the first numeric column that is not
809    an ID or datetime column.
810    If none may be found, return `None`.
811
812    Parameters
813    ----------
814    debug: bool, default False:
815        Verbosity toggle.
816
817    Returns
818    -------
819    Either a string or `None`.
820    """
821    if debug:
822        dprint('Attempting to determine the value column...')
823    try:
824        val_name = self.get_columns('value')
825    except Exception:
826        val_name = None
827    if val_name is not None:
828        if debug:
829            dprint(f"Value column: {val_name}")
830        return val_name
831
832    cols = self.columns
833    if cols is None:
834        if debug:
835            dprint('No columns could be determined. Returning...')
836        return None
837    try:
838        dt_name = self.get_columns('datetime', error=False)
839    except Exception:
840        dt_name = None
841    try:
842        id_name = self.get_columns('id', errors=False)
843    except Exception:
844        id_name = None
845
846    if debug:
847        dprint(f"dt_name: {dt_name}")
848        dprint(f"id_name: {id_name}")
849
850    cols_types = self.get_columns_types(debug=debug)
851    if cols_types is None:
852        return None
853    if debug:
854        dprint(f"cols_types: {cols_types}")
855    if dt_name is not None:
856        cols_types.pop(dt_name, None)
857    if id_name is not None:
858        cols_types.pop(id_name, None)
859
860    candidates = []
861    candidate_keywords = {'float', 'double', 'precision', 'int', 'numeric',}
862    for search_term in candidate_keywords:
863        for col, typ in cols_types.items():
864            if search_term in typ.lower():
865                candidates.append(col)
866                break
867    if not candidates:
868        if debug:
869            dprint("No value column could be determined.")
870        return None
871
872    return candidates[0]

Return the name of the value column if it's defined, otherwise make an educated guess. If not set in the columns dictionary, return the first numeric column that is not an ID or datetime column. If none may be found, return None.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
  • Either a string or None.
parents: List[Pipe] 
875@property
876def parents(self) -> List[mrsm.Pipe]:
877    """
878    Return a list of `meerschaum.Pipe` objects to be designated as parents.
879    """
880    _cached_parents = self.__dict__.get('_parents', None)
881    if _cached_parents is not None:
882        return _cached_parents
883
884    from meerschaum.utils.pipes import get_pipe_from_value
885    base_params = self.get_parameters()
886    key = 'parents' if 'parents' in base_params else 'parent'
887    parents_refs = base_params.get(key, None) or []
888    if isinstance(parents_refs, str) or isinstance(parents_refs, dict):
889        parents_refs = [parents_refs]
890
891    if not parents_refs:
892        return []
893
894    self._parents = [get_pipe_from_value(val, _pipe=self) for val in parents_refs]
895    return self._parents

Return a list of meerschaum.Pipe objects to be designated as parents.

parent: Pipe | None 
898@property
899def parent(self) -> Union[mrsm.Pipe, None]:
900    """
901    Return the first pipe in `self.parents` or `None`.
902    """
903    _parents = self.parents
904    if not _parents:
905        return None
906
907    return _parents[0]

Return the first pipe in self.parents or None.

children: List[Pipe] 
943@property
944def children(self) -> List[mrsm.Pipe]:
945    """
946    Return a list of `meerschaum.Pipe` objects to be designated as children.
947    """
948    _cached_children = self.__dict__.get('_children', None)
949    if _cached_children is not None:
950        return _cached_children
951
952    from meerschaum.utils.pipes import get_pipe_from_value
953    base_params = self.get_parameters()
954    key = 'children' if 'children' in base_params else 'child'
955    children_refs = base_params.get(key, None) or []
956    if isinstance(children_refs, str) or isinstance(children_refs, dict):
957        children_refs = [children_refs]
958
959    if not children_refs:
960        return []
961
962    self._children = [get_pipe_from_value(val, _pipe=self) for val in children_refs]
963    return self._children

Return a list of meerschaum.Pipe objects to be designated as children.

child: Pipe | None 
966@property
967def child(self) -> mrsm.Pipe | None:
968    """
969    Return the first pipe in `self.children` or None.
970    """
971    _children = self.children
972    if not _children:
973        return None
974
975    return _children[0]

Return the first pipe in self.children or None.

reference: Pipe | None 
1035@property
1036def reference(self) -> mrsm.Pipe | None:
1037    """
1038    Return the first pipe in `self.references` or None.
1039    """
1040    _references = self.references
1041    if not _references:
1042        return None
1043
1044    return _references[0]

Return the first pipe in self.references or None.

references: List[Pipe] 
1012@property
1013def references(self) -> List[mrsm.Pipe]:
1014    """
1015    Return a list of `meerschaum.Pipe` objects to be designated as references.
1016    """
1017    _cached_references = self.__dict__.get('_references', None)
1018    if _cached_references is not None:
1019        return _cached_references
1020
1021    from meerschaum.utils.pipes import get_pipe_from_value
1022    base_params = self.get_parameters(apply_symlinks=False)
1023    key = 'references' if 'references' in base_params else 'reference'
1024    refs = base_params.get(key, None) or []
1025    if isinstance(refs, str) or isinstance(refs, dict):
1026        refs = [refs]
1027
1028    if not refs:
1029        return []
1030
1031    self._references = [get_pipe_from_value(val, _pipe=self) for val in refs]
1032    return self._references

Return a list of meerschaum.Pipe objects to be designated as references.

target: str 
1082@property
1083def target(self) -> str:
1084    """
1085    The target table name.
1086    Registered pipes do not memoize guessed defaults while metadata failures are outstanding.
1087    You can set the target name under on of the following keys
1088    (checked in this order):
1089      - `target`
1090      - `target_name`
1091      - `target_table`
1092      - `target_table_name`
1093    """
1094    cached_target = self.__dict__.get('_target', None)
1095    if cached_target:
1096        return cached_target
1097
1098    load_failures = self.__dict__.get('_attributes_load_failures', 0)
1099    params = self.parameters
1100    target_val = params.get('target', None)
1101    if target_val:
1102        self.__dict__['_target'] = target_val
1103        return target_val
1104
1105    core = self._core_pipe
1106    default_target = self._target_legacy()
1107    default_targets = {default_target}
1108    ### The Rust name rules (`target_name` / `target_table` / ... keys, the legacy default),
1109    ### over the parameters resolved here (so cross-pipe templates are honored).
1110    _target = core.target(None, dict(params))
1111
1112    if self.instance_connector.type == 'sql':
1113        truncated_target = core.target(self.instance_connector.flavor, dict(params))
1114        default_targets.add(truncated_target)
1115        warned_target = self.__dict__.get('_warned_target', False)
1116        if truncated_target != _target and not warned_target:
1117            if self.instance_connector.flavor not in ('oracle', 'mysql', 'mariadb'):
1118                warn(
1119                    f"The target '{_target}' is too long for '{self.instance_connector.flavor}', "
1120                    + f"will use {truncated_target} instead."
1121                )
1122            self.__dict__['_warned_target'] = True
1123            _target = truncated_target
1124
1125    if _target not in default_targets:
1126        self.target = _target
1127
1128    # Parameter resolution may retry a failed load; its first snapshot is still stale.
1129    resolved_cleanly = load_failures == self.__dict__.get('_attributes_load_failures', 0)
1130    if (params and resolved_cleanly) or self.temporary or self.id is None:
1131        self.__dict__['_target'] = _target
1132    return _target

The target table name. Registered pipes do not memoize guessed defaults while metadata failures are outstanding. You can set the target name under on of the following keys (checked in this order):

  • target
  • target_name
  • target_table
  • target_table_name
def guess_datetime(self) -> str | None: 
1153def guess_datetime(self) -> Union[str, None]:
1154    """
1155    Try to determine a pipe's datetime column.
1156    """
1157    _dtypes = self.dtypes
1158
1159    ### Abort if the user explictly disallows a datetime index.
1160    if 'datetime' in _dtypes:
1161        if _dtypes['datetime'] is None:
1162            return None
1163
1164    from meerschaum.utils.dtypes import are_dtypes_equal
1165    dt_cols = [
1166        col
1167        for col, typ in _dtypes.items()
1168        if are_dtypes_equal(typ, 'datetime')
1169    ]
1170    if not dt_cols:
1171        return None
1172    return dt_cols[0]

Try to determine a pipe's datetime column.

precision: Dict[str, str | int] 
1317@property
1318def precision(self) -> Dict[str, Union[str, int]]:
1319    """
1320    Return the configured or detected precision.
1321    """
1322    return self.get_precision(debug=self.debug)

Return the configured or detected precision.

def get_precision(self, debug: bool = False) -> Dict[str, str | int]: 
1228def get_precision(self, debug: bool = False) -> Dict[str, Union[str, int]]:
1229    """
1230    Return the timestamp precision unit and interval for the `datetime` axis.
1231    """
1232    from meerschaum.utils.dtypes import (
1233        MRSM_PRECISION_UNITS_SCALARS,
1234        MRSM_PRECISION_UNITS_ALIASES,
1235        MRSM_PD_DTYPES,
1236        are_dtypes_equal,
1237    )
1238    from meerschaum._internal.static import STATIC_CONFIG
1239
1240    _precision = self._get_cached_value('precision', debug=debug)
1241    if _precision:
1242        if debug:
1243            dprint(f"Returning cached precision: {_precision}")
1244        return _precision
1245
1246    parameters = self.parameters
1247    _precision = parameters.get('precision', {})
1248    if isinstance(_precision, str):
1249        _precision = {'unit': _precision}
1250    default_precision_unit = STATIC_CONFIG['dtypes']['datetime']['default_precision_unit']
1251
1252    if not _precision:
1253
1254        dt_col = parameters.get('columns', {}).get('datetime', None)
1255        if not dt_col and self.autotime:
1256            dt_col = mrsm.get_config('pipes', 'autotime', 'column_name_if_datetime_missing')
1257        if not dt_col:
1258            if debug:
1259                dprint(f"No datetime axis, returning default precision '{default_precision_unit}'.")
1260            return {'unit': default_precision_unit}
1261
1262        dt_typ = self.dtypes.get(dt_col, 'datetime')
1263        if are_dtypes_equal(dt_typ, 'datetime'):
1264            if dt_typ == 'datetime':
1265                dt_typ = MRSM_PD_DTYPES['datetime']
1266                if debug:
1267                    dprint(f"Datetime type is `datetime`, assuming {dt_typ} precision.")
1268
1269            _precision = {
1270                'unit': (
1271                    dt_typ
1272                    .split('[', maxsplit=1)[-1]
1273                    .split(',', maxsplit=1)[0]
1274                    .split(' ', maxsplit=1)[0]
1275                ).rstrip(']')
1276            }
1277
1278            if debug:
1279                dprint(f"Extracted precision '{_precision['unit']}' from type '{dt_typ}'.")
1280
1281        elif are_dtypes_equal(dt_typ, 'int'):
1282            _precision = {
1283                'unit': (
1284                    'second'
1285                    if '32' in dt_typ
1286                    else default_precision_unit
1287                )
1288            }
1289        elif are_dtypes_equal(dt_typ, 'date'):
1290            if debug:
1291                dprint("Datetime axis is 'date', falling back to 'day' precision.")
1292            _precision = {'unit': 'day'}
1293
1294    precision_unit = _precision.get('unit', default_precision_unit)
1295    precision_interval = _precision.get('interval', None)
1296    true_precision_unit = MRSM_PRECISION_UNITS_ALIASES.get(precision_unit, precision_unit)
1297    if true_precision_unit is None:
1298        if debug:
1299            dprint(f"No precision could be determined, falling back to '{default_precision_unit}'.")
1300        true_precision_unit = default_precision_unit
1301
1302    if true_precision_unit not in MRSM_PRECISION_UNITS_SCALARS:
1303        from meerschaum.utils.misc import items_str
1304        raise ValueError(
1305            f"Invalid precision unit '{true_precision_unit}'.\n"
1306            "Accepted values are "
1307            f"{items_str(list(MRSM_PRECISION_UNITS_SCALARS) + list(MRSM_PRECISION_UNITS_ALIASES))}."
1308        )
1309
1310    _precision = {'unit': true_precision_unit}
1311    if precision_interval:
1312        _precision['interval'] = precision_interval
1313    self._cache_value('precision', _precision, debug=debug)
1314    return self._precision

Return the timestamp precision unit and interval for the datetime axis.

def show( self, nopretty: bool = False, debug: bool = False, **kw) -> Tuple[bool, str]: 
12def show(
13    self,
14    nopretty: bool = False,
15    debug: bool = False,
16    **kw
17) -> SuccessTuple:
18    """
19    Show attributes of a Pipe.
20
21    Parameters
22    ----------
23    nopretty: bool, default False
24        If `True`, simply print the JSON of the pipe's attributes.
25
26    debug: bool, default False
27        Verbosity toggle.
28
29    Returns
30    -------
31    A `SuccessTuple` of success, message.
32
33    """
34    import json
35    from meerschaum.utils.formatting import (
36        pprint, make_header, ANSI, highlight_pipes, fill_ansi, get_console,
37    )
38    from meerschaum.utils.packages import import_rich, attempt_import
39    from meerschaum.utils.warnings import info
40    attributes_json = json.dumps(self.attributes)
41    if not nopretty:
42        _to_print = f"Attributes for {self}:"
43        if ANSI:
44            _to_print = fill_ansi(highlight_pipes(make_header(_to_print)), 'magenta')
45            print(_to_print)
46            rich = import_rich()
47            rich_json = attempt_import('rich.json')
48            get_console().print(rich_json.JSON(attributes_json))
49        else:
50            print(_to_print)
51    else:
52        print(attributes_json)
53
54    return True, "Success"

Show attributes of a Pipe.

Parameters
  • nopretty (bool, default False): If True, simply print the JSON of the pipe's attributes.
  • debug (bool, default False): Verbosity toggle.
Returns
def edit( self, patch: bool = False, interactive: bool = False, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
 21def edit(
 22    self,
 23    patch: bool = False,
 24    interactive: bool = False,
 25    debug: bool = False,
 26    **kw: Any
 27) -> SuccessTuple:
 28    """
 29    Edit a Pipe's configuration.
 30
 31    Parameters
 32    ----------
 33    patch: bool, default False
 34        If `patch` is True, update parameters by cascading rather than overwriting.
 35    interactive: bool, default False
 36        If `True`, open an editor for the user to make changes to the pipe's YAML file.
 37    debug: bool, default False
 38        Verbosity toggle.
 39
 40    Returns
 41    -------
 42    A `SuccessTuple` of success, message.
 43
 44    """
 45    from meerschaum.utils.venv import Venv
 46    from meerschaum.connectors import get_connector_plugin
 47
 48    if self.temporary:
 49        return False, "Cannot edit pipes created with `temporary=True` (read-only)."
 50
 51    self._invalidate_cache(hard=True, debug=debug)
 52
 53    if hasattr(self, '_symlinks'):
 54        from meerschaum.utils.misc import get_val_from_dict_path, set_val_in_dict_path
 55        for path, vals in self._symlinks.items():
 56            current_val = get_val_from_dict_path(self.parameters, path)
 57            if current_val == vals['substituted']:
 58                set_val_in_dict_path(self.parameters, path, vals['original'])
 59
 60    if not interactive:
 61        ### Refuse an index column which cannot be sorted or deduplicated.
 62        indices_success, indices_msg = self.validate_indices(debug=debug)
 63        if not indices_success:
 64            return indices_success, indices_msg
 65
 66        ### A `parameters` lock allows changing `locks` only (the core checks its own instances).
 67        refusal = self._edit_lock_refusal(patch=patch, debug=debug)
 68        if refusal:
 69            return False, refusal
 70
 71        with Venv(get_connector_plugin(self.instance_connector)):
 72            return self.instance_connector.edit_pipe(self, patch=patch, debug=debug, **kw)
 73
 74    import meerschaum.config.paths as paths
 75    from meerschaum.utils.misc import edit_file
 76    parameters_filename = str(self) + '.yaml'
 77    parameters_path = paths.PIPES_CACHE_RESOURCES_PATH / parameters_filename
 78
 79    from meerschaum.utils.yaml import yaml
 80
 81    edit_text = f"Edit the parameters for {self}"
 82    edit_top = '#' * (len(edit_text) + 4)
 83    edit_header = edit_top + f'\n# {edit_text} #\n' + edit_top + '\n\n'
 84
 85    from meerschaum.config import get_config
 86    parameters = dict(get_config('pipes', 'parameters', patch=True))
 87    from meerschaum.config._patch import apply_patch_to_config
 88    raw_parameters = self.attributes.get('parameters', {})
 89    parameters = apply_patch_to_config(parameters, raw_parameters)
 90
 91    ### write parameters to yaml file
 92    with open(parameters_path, 'w+') as f:
 93        f.write(edit_header)
 94        yaml.dump(parameters, stream=f, sort_keys=False)
 95
 96    ### only quit editing if yaml is valid
 97    editing = True
 98    while editing:
 99        edit_file(parameters_path)
100        try:
101            with open(parameters_path, 'r') as f:
102                file_parameters = yaml.load(f.read())
103        except Exception as e:
104            from meerschaum.utils.warnings import warn
105            warn(f"Invalid format defined for '{self}':\n\n{e}")
106            input(f"Press [Enter] to correct the configuration for '{self}': ")
107        else:
108            editing = False
109
110    self.parameters = file_parameters
111
112    if debug:
113        from meerschaum.utils.formatting import pprint
114        pprint(self.parameters)
115
116    refusal = self._edit_lock_refusal(patch=patch, debug=debug)
117    if refusal:
118        return False, refusal
119
120    with Venv(get_connector_plugin(self.instance_connector)):
121        return self.instance_connector.edit_pipe(self, patch=patch, debug=debug, **kw)

Edit a Pipe's configuration.

Parameters
  • patch (bool, default False): If patch is True, update parameters by cascading rather than overwriting.
  • interactive (bool, default False): If True, open an editor for the user to make changes to the pipe's YAML file.
  • debug (bool, default False): Verbosity toggle.
Returns
def edit_definition( self, yes: bool = False, noask: bool = False, force: bool = False, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
124def edit_definition(
125    self,
126    yes: bool = False,
127    noask: bool = False,
128    force: bool = False,
129    debug : bool = False,
130    **kw : Any
131) -> SuccessTuple:
132    """
133    Edit a pipe's definition file and update its configuration.
134    **NOTE:** This function is interactive and should not be used in automated scripts!
135
136    Returns
137    -------
138    A `SuccessTuple` of success, message.
139
140    """
141    if self.temporary:
142        return False, "Cannot edit pipes created with `temporary=True` (read-only)."
143
144    from meerschaum.connectors import instance_types
145    if (self.connector is None or isinstance(self.connector, str)) or self.connector.type not in instance_types:
146        return self.edit(interactive=True, debug=debug, **kw)
147
148    import json
149    from meerschaum.utils.warnings import info, warn
150    from meerschaum.utils.debug import dprint
151    from meerschaum.config._patch import apply_patch_to_config
152    from meerschaum.utils.misc import edit_file
153
154    _parameters = self.parameters
155    if 'fetch' not in _parameters:
156        _parameters['fetch'] = {}
157
158    def _edit_api():
159        from meerschaum.utils.prompt import prompt, yes_no
160        info(
161            f"Please enter the keys of the source pipe from '{self.connector}'.\n" +
162            "Type 'None' for None, or empty when there is no default. Press [CTRL+C] to skip."
163        )
164
165        _keys = { 'connector_keys' : None, 'metric_key' : None, 'location_key' : None }
166        for k in _keys:
167            _keys[k] = _parameters['fetch'].get(k, None)
168
169        for k, v in _keys.items():
170            try:
171                _keys[k] = prompt(k.capitalize().replace('_', ' ') + ':', icon=True, default=v)
172            except KeyboardInterrupt:
173                continue
174            if _keys[k] in ('', 'None', '\'None\'', '[None]'):
175                _keys[k] = None
176
177        _parameters['fetch'] = apply_patch_to_config(_parameters['fetch'], _keys)
178
179        info("You may optionally specify additional filter parameters as JSON.")
180        print("  Parameters are translated into a 'WHERE x AND y' clause, and lists are IN clauses.")
181        print("  For example, the following JSON would correspond to 'WHERE x = 1 AND y IN (2, 3)':")
182        print(json.dumps({'x': 1, 'y': [2, 3]}, indent=2, separators=(',', ': ')))
183        if force or yes_no(
184            "Would you like to add additional filter parameters?",
185            yes=yes, noask=noask
186        ):
187            import meerschaum.config.paths as paths
188            definition_filename = str(self) + '.json'
189            definition_path = paths.PIPES_CACHE_RESOURCES_PATH / definition_filename
190            try:
191                definition_path.touch()
192                with open(definition_path, 'w+') as f:
193                    json.dump(_parameters.get('fetch', {}).get('params', {}), f, indent=2)
194            except Exception as e:
195                return False, f"Failed writing file '{definition_path}':\n" + str(e)
196
197            _params = None
198            while True:
199                edit_file(definition_path)
200                try:
201                    with open(definition_path, 'r') as f:
202                        _params = json.load(f)
203                except Exception as e:
204                    warn(f'Failed to read parameters JSON:\n{e}', stack=False)
205                    if force or yes_no(
206                        "Would you like to try again?\n  "
207                        + "If not, the parameters JSON file will be ignored.",
208                        noask=noask, yes=yes
209                    ):
210                        continue
211                    _params = None
212                break
213            if _params is not None:
214                if 'fetch' not in _parameters:
215                    _parameters['fetch'] = {}
216                _parameters['fetch']['params'] = _params
217
218        self.parameters = _parameters
219        return True, "Success"
220
221    def _edit_sql():
222        import textwrap
223        import meerschaum.config.paths as paths
224        from meerschaum.utils.misc import edit_file
225        definition_filename = str(self) + '.sql'
226        definition_path = paths.PIPES_CACHE_RESOURCES_PATH / definition_filename
227
228        sql_definition = _parameters['fetch'].get('definition', None)
229        if sql_definition is None:
230            sql_definition = ''
231        sql_definition = textwrap.dedent(sql_definition).lstrip()
232
233        try:
234            definition_path.touch()
235            with open(definition_path, 'w+') as f:
236                f.write(sql_definition)
237        except Exception as e:
238            return False, f"Failed writing file '{definition_path}':\n" + str(e)
239
240        edit_file(definition_path)
241        try:
242            with open(definition_path, 'r', encoding='utf-8') as f:
243                file_definition = f.read()
244        except Exception as e:
245            return False, f"Failed reading file '{definition_path}':\n" + str(e)
246
247        if sql_definition == file_definition:
248            return False, f"No changes made to definition for {self}."
249
250        if ' ' not in file_definition:
251            return False, f"Invalid SQL definition for {self}."
252
253        if debug:
254            dprint("Read SQL definition:\n\n" + file_definition)
255        _parameters['fetch']['definition'] = file_definition
256        self.parameters = _parameters
257        return True, "Success"
258
259    locals()['_edit_' + str(self.connector.type)]()
260    return self.edit(interactive=False, debug=debug, **kw)

Edit a pipe's definition file and update its configuration. NOTE: This function is interactive and should not be used in automated scripts!

Returns
def update(self, *args, **kw) -> Tuple[bool, str]: 
13def update(self, *args, **kw) -> SuccessTuple:
14    """
15    Update a pipe's parameters in its instance.
16    """
17    kw['interactive'] = False
18    return self.edit(*args, **kw)

Update a pipe's parameters in its instance.

def sync( self, df: pandas.DataFrame | Dict[str, List[Any]] | List[Dict[str, Any]] | str | meerschaum.core.Pipe._sync.InferFetch = <class 'meerschaum.core.Pipe._sync.InferFetch'>, begin: datetime.datetime | int | str | None = '', end: datetime.datetime | int | None = None, force: bool = False, retries: int = 10, min_seconds: int = 1, check_existing: bool = True, enforce_dtypes: bool = True, blocking: bool = True, workers: int | None = None, callback: Callable[[Tuple[bool, str]], Any] | None = None, error_callback: Callable[[Exception], Any] | None = None, chunksize: int | None = -1, sync_chunks: bool = True, parents: bool | int | None = None, children: bool | int | None = None, replace: bool = False, debug: bool = False, _inplace: bool = True, **kw: Any) -> Tuple[bool, str]: 
736def sync(
737    self,
738    df: Union[
739        pd.DataFrame,
740        Dict[str, List[Any]],
741        List[Dict[str, Any]],
742        str,
743        InferFetch
744    ] = InferFetch,
745    begin: Union[datetime, int, str, None] = '',
746    end: Union[datetime, int, None] = None,
747    force: bool = False,
748    retries: int = 10,
749    min_seconds: int = 1,
750    check_existing: bool = True,
751    enforce_dtypes: bool = True,
752    blocking: bool = True,
753    workers: Optional[int] = None,
754    callback: Optional[Callable[[Tuple[bool, str]], Any]] = None,
755    error_callback: Optional[Callable[[Exception], Any]] = None,
756    chunksize: Optional[int] = -1,
757    sync_chunks: bool = True,
758    parents: Union[bool, int, None] = None,
759    children: Union[bool, int, None] = None,
760    replace: bool = False,
761    debug: bool = False,
762    _inplace: bool = True,
763    **kw: Any
764) -> SuccessTuple:
765    """
766    Fetch new data from the source and update the pipe's table with new data.
767    
768    Get new remote data via fetch, get existing data in the same time period,
769    and merge the two, only keeping the unseen data.
770
771    Parameters
772    ----------
773    df: Union[None, pd.DataFrame, Dict[str, List[Any]]], default None
774        An optional DataFrame to sync into the pipe. Defaults to `None`.
775        If `df` is a string, it will be parsed via `meerschaum.utils.dataframe.parse_simple_lines()`.
776
777    begin: Union[datetime, int, str, None], default ''
778        Optionally specify the earliest datetime to search for data.
779
780    end: Union[datetime, int, str, None], default None
781        Optionally specify the latest datetime to search for data.
782
783    force: bool, default False
784        If `True`, keep trying to sync untul `retries` attempts.
785
786    retries: int, default 10
787        If `force`, how many attempts to try syncing before declaring failure.
788
789    min_seconds: Union[int, float], default 1
790        If `force`, how many seconds to sleep between retries. Defaults to `1`.
791
792    check_existing: bool, default True
793        If `True`, pull and diff with existing data from the pipe.
794
795    enforce_dtypes: bool, default True
796        If `True`, enforce dtypes on incoming data.
797        Set this to `False` if the incoming rows are expected to be of the correct dtypes.
798
799    blocking: bool, default True
800        If `True`, wait for sync to finish and return its result, otherwise
801        asyncronously sync (oxymoron?) and return success. Defaults to `True`.
802        Only intended for specific scenarios.
803
804    workers: Optional[int], default None
805        If provided and the instance connector is thread-safe
806        (`pipe.instance_connector.IS_THREAD_SAFE is True`),
807        limit concurrent sync to this many threads.
808
809    callback: Optional[Callable[[Tuple[bool, str]], Any]], default None
810        Callback function which expects a SuccessTuple as input.
811        Only applies when `blocking=False`.
812
813    error_callback: Optional[Callable[[Exception], Any]], default None
814        Callback function which expects an Exception as input.
815        Only applies when `blocking=False`.
816
817    chunksize: int, default -1
818        Specify the number of rows to sync per chunk.
819        If `-1`, resort to system configuration (default is `900`).
820        A `chunksize` of `None` will sync all rows in one transaction.
821
822    sync_chunks: bool, default True
823        If possible, sync chunks while fetching them into memory.
824
825    parents: Union[bool, int, None], default None
826        If `True`, sync every ancestor of this pipe first, in dependency order (the implicit
827        pipe graph, including `{{ Pipe() }}` references). An `int` syncs that many generations
828        (`parents=1` is the direct parents). Runs through the Rust Syncing Engine, which skips
829        parents and children that are already caught up. A bare `sync()` (the default) never
830        uses the engine and behaves exactly like v4.
831
832    children: Union[bool, int, None], default None
833        If `True`, sync this pipe, then propagate to every descendant. An `int` limits the
834        generations. Combine with `parents` for the whole connected slice.
835        See `meerschaum.Pipeline` to plan the run first and read the run history.
836
837    replace: bool, default False
838        If `True`, the rows you sync replace the pipe's rows in `[begin, end)` instead of being
839        merged into them: in one transaction, the rows of that window (matching `params`, if
840        given) are deleted and the new rows inserted, so rows missing from the new data are
841        removed. When you pass data, `begin` defaults to the earliest datetime in the data and
842        `end` to just after the latest one, and rows outside `[begin, end)` are not written
843        (the message says how many were skipped). An empty frame with `begin` and `end` clears
844        the window. Without data (a fetch), the window is the fetch's own `[begin, end)`; set
845        `fetch: {replace: true}` in the parameters to do this on every sync. A pipe without a
846        datetime column has one window: the whole table. Valkey instances clear the window
847        then write the rows (no transaction).
848
849    on_event: Optional[Callable[[Dict[str, Any]], Any]], default None
850        Only with `parents` or `children`: called on this thread with every event of the engine
851        run as it happens (`run_started`, `node_started`, `chunk_finished`, `node_finished`,
852        ..., `run_finished`), each a `dict` with `event`, `run_id`, `seq`, `ts`, `pipe_ref` and
853        the event's own fields. An exception it raises is warned about and the run continues
854        (see `meerschaum.Pipeline.sync`).
855
856    debug: bool, default False
857        Verbosity toggle. Defaults to False.
858
859    Returns
860    -------
861    A `SyncResult`: a `SuccessTuple` of success (`bool`) and message (`str`) that also carries
862    the structured fields of the sync (`inserted`, `updated`, `upserted`, `deleted`,
863    `changed_intervals`, `duration`, `begin` and `end`; `None` where the sync did not
864    report them).
865    With `parents` or `children`, a `meerschaum.core.Pipeline.RunResult` (a `SuccessTuple`
866    with the run's `run_id`, `plan`, `outcomes` and `events()`).
867    """
868    if replace:
869        kw['replace'] = True
870    if parents or children:
871        from meerschaum.core.Pipeline import sync_with_graph
872        return sync_with_graph(
873            self,
874            df,
875            parents=parents,
876            children=children,
877            begin=begin,
878            end=end,
879            force=force,
880            retries=retries,
881            min_seconds=min_seconds,
882            check_existing=check_existing,
883            enforce_dtypes=enforce_dtypes,
884            workers=workers,
885            chunksize=chunksize,
886            debug=debug,
887            **kw
888        )
889
890    from meerschaum import _core
891    return _sync_native(
892        self,
893        _core.sync_pipe,
894        df,
895        begin=begin,
896        end=end,
897        check_existing=check_existing,
898        enforce_dtypes=enforce_dtypes,
899        chunksize=chunksize,
900        sync_chunks=sync_chunks,
901        force=force,
902        retries=retries,
903        min_seconds=min_seconds,
904        blocking=blocking,
905        workers=workers,
906        callback=callback,
907        error_callback=error_callback,
908        debug=debug,
909        inplace=(
910            _inplace
911            and bool(mrsm.get_config('system', 'experimental', 'inplace_sync'))
912        ),
913        kw=kw,
914        replace=bool(replace),
915    )

Fetch new data from the source and update the pipe's table with new data.

Get new remote data via fetch, get existing data in the same time period, and merge the two, only keeping the unseen data.

Parameters
  • df (Union[None, pd.DataFrame, Dict[str, List[Any]]], default None): An optional DataFrame to sync into the pipe. Defaults to None. If df is a string, it will be parsed via meerschaum.utils.dataframe.parse_simple_lines().
  • begin (Union[datetime, int, str, None], default ''): Optionally specify the earliest datetime to search for data.
  • end (Union[datetime, int, str, None], default None): Optionally specify the latest datetime to search for data.
  • force (bool, default False): If True, keep trying to sync untul retries attempts.
  • retries (int, default 10): If force, how many attempts to try syncing before declaring failure.
  • min_seconds (Union[int, float], default 1): If force, how many seconds to sleep between retries. Defaults to 1.
  • check_existing (bool, default True): If True, pull and diff with existing data from the pipe.
  • enforce_dtypes (bool, default True): If True, enforce dtypes on incoming data. Set this to False if the incoming rows are expected to be of the correct dtypes.
  • blocking (bool, default True): If True, wait for sync to finish and return its result, otherwise asyncronously sync (oxymoron?) and return success. Defaults to True. Only intended for specific scenarios.
  • workers (Optional[int], default None): If provided and the instance connector is thread-safe (pipe.instance_connector.IS_THREAD_SAFE is True), limit concurrent sync to this many threads.
  • callback (Optional[Callable[[Tuple[bool, str]], Any]], default None): Callback function which expects a SuccessTuple as input. Only applies when blocking=False.
  • error_callback (Optional[Callable[[Exception], Any]], default None): Callback function which expects an Exception as input. Only applies when blocking=False.
  • chunksize (int, default -1): Specify the number of rows to sync per chunk. If -1, resort to system configuration (default is 900). A chunksize of None will sync all rows in one transaction.
  • sync_chunks (bool, default True): If possible, sync chunks while fetching them into memory.
  • parents (Union[bool, int, None], default None): If True, sync every ancestor of this pipe first, in dependency order (the implicit pipe graph, including {{ Pipe() }} references). An int syncs that many generations (parents=1 is the direct parents). Runs through the Rust Syncing Engine, which skips parents and children that are already caught up. A bare sync() (the default) never uses the engine and behaves exactly like v4.
  • children (Union[bool, int, None], default None): If True, sync this pipe, then propagate to every descendant. An int limits the generations. Combine with parents for the whole connected slice. See meerschaum.Pipeline to plan the run first and read the run history.
  • replace (bool, default False): If True, the rows you sync replace the pipe's rows in [begin, end) instead of being merged into them: in one transaction, the rows of that window (matching params, if given) are deleted and the new rows inserted, so rows missing from the new data are removed. When you pass data, begin defaults to the earliest datetime in the data and end to just after the latest one, and rows outside [begin, end) are not written (the message says how many were skipped). An empty frame with begin and end clears the window. Without data (a fetch), the window is the fetch's own [begin, end); set fetch: {replace: true} in the parameters to do this on every sync. A pipe without a datetime column has one window: the whole table. Valkey instances clear the window then write the rows (no transaction).
  • on_event (Optional[Callable[[Dict[str, Any]], Any]], default None): Only with parents or children: called on this thread with every event of the engine run as it happens (run_started, node_started, chunk_finished, node_finished, ..., run_finished), each a dict with event, run_id, seq, ts, pipe_ref and the event's own fields. An exception it raises is warned about and the run continues (see meerschaum.Pipeline.sync).
  • debug (bool, default False): Verbosity toggle. Defaults to False.
Returns
def get_sync_time( self, params: Dict[str, Any] | None = None, newest: bool = True, apply_backtrack_interval: bool = False, remote: bool = False, round_down: bool = False, silent: bool = False, debug: bool = False) -> datetime.datetime | int | None: 
918def get_sync_time(
919    self,
920    params: Optional[Dict[str, Any]] = None,
921    newest: bool = True,
922    apply_backtrack_interval: bool = False,
923    remote: bool = False,
924    round_down: bool = False,
925    silent: bool = False,
926    debug: bool = False
927) -> Union['datetime', int, None]:
928    """
929    Get the most recent datetime value for a Pipe.
930
931    Parameters
932    ----------
933    params: Optional[Dict[str, Any]], default None
934        Dictionary to build a WHERE clause for a specific column.
935        See `meerschaum.utils.sql.build_where`.
936
937    newest: bool, default True
938        If `True`, get the most recent datetime (honoring `params`).
939        If `False`, get the oldest datetime (`ASC` instead of `DESC`).
940
941    apply_backtrack_interval: bool, default False
942        If `True`, subtract the backtrack interval from the sync time.
943
944    remote: bool, default False
945        If `True` and the instance connector supports it, return the sync time
946        for the remote table definition.
947
948    round_down: bool, default False
949        If `True`, round down the datetime value to the nearest minute.
950
951    silent: bool, default False
952        If `True`, return `None` instead of raising when the database cannot be read
953        (Meerschaum 4's behavior).
954
955    debug: bool, default False
956        Verbosity toggle.
957
958    Returns
959    -------
960    A `datetime` or int, if the pipe exists, otherwise `None` (a pipe that was never synced
961    has no sync time). Raises the database error if the instance cannot be read.
962
963    """
964    from meerschaum import _core
965    from meerschaum.core.Pipe._data import _rust_context
966    try:
967        context = _rust_context(self, debug=debug)
968        return _core.pipe_get_sync_time(
969            self,
970            context,
971            params=params,
972            newest=newest,
973            apply_backtrack_interval=apply_backtrack_interval,
974            remote=remote,
975            round_down=round_down,
976            debug=debug,
977        )
978    except Exception:
979        if silent:
980            return None
981        raise

Get the most recent datetime value for a Pipe.

Parameters
  • params (Optional[Dict[str, Any]], default None): Dictionary to build a WHERE clause for a specific column. See meerschaum.utils.sql.build_where.
  • newest (bool, default True): If True, get the most recent datetime (honoring params). If False, get the oldest datetime (ASC instead of DESC).
  • apply_backtrack_interval (bool, default False): If True, subtract the backtrack interval from the sync time.
  • remote (bool, default False): If True and the instance connector supports it, return the sync time for the remote table definition.
  • round_down (bool, default False): If True, round down the datetime value to the nearest minute.
  • silent (bool, default False): If True, return None instead of raising when the database cannot be read (Meerschaum 4's behavior).
  • debug (bool, default False): Verbosity toggle.
Returns
  • A datetime or int, if the pipe exists, otherwise None (a pipe that was never synced
  • has no sync time). Raises the database error if the instance cannot be read.
def exists(self, debug: bool = False) -> bool: 
 984def exists(
 985    self,
 986    debug: bool = False
 987) -> bool:
 988    """
 989    See if a Pipe's table exists.
 990
 991    Parameters
 992    ----------
 993    debug: bool, default False
 994        Verbosity toggle.
 995
 996    Returns
 997    -------
 998    A `bool` corresponding to whether a pipe's underlying table exists.
 999
1000    """
1001    from meerschaum.utils.venv import Venv
1002    from meerschaum.connectors import get_connector_plugin
1003    from meerschaum.utils.debug import dprint
1004    from meerschaum.utils.dtypes import get_current_timestamp
1005    now = get_current_timestamp('ms', as_int=True) / 1000
1006    cache_seconds = mrsm.get_config('pipes', 'sync', 'exists_cache_seconds')
1007
1008    _exists = self._get_cached_value('_exists', debug=debug)
1009    if _exists:
1010        exists_timestamp = self._get_cached_value('_exists_timestamp', debug=debug)
1011        if exists_timestamp is not None:
1012            delta = now - exists_timestamp
1013            if delta < cache_seconds:
1014                if debug:
1015                    dprint(f"Returning cached `exists` for {self} ({round(delta, 2)} seconds old).")
1016                return _exists
1017
1018    with Venv(get_connector_plugin(self.instance_connector)):
1019        _exists = (
1020            self.instance_connector.pipe_exists(pipe=self, debug=debug)
1021            if hasattr(self.instance_connector, 'pipe_exists')
1022            else False
1023        )
1024
1025    self._cache_value('_exists', _exists, debug=debug)
1026    self._cache_value('_exists_timestamp', now, debug=debug)
1027    return _exists

See if a Pipe's table exists.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
  • A bool corresponding to whether a pipe's underlying table exists.
def filter_existing( self, df: pandas.DataFrame, safe_copy: bool = True, date_bound_only: bool = False, include_unchanged_columns: bool = False, enforce_dtypes: bool = False, chunksize: int | None = -1, debug: bool = False, as_polars: bool | None = None, as_pandas: bool | None = None, **kw) -> Tuple[pandas.DataFrame, pandas.DataFrame, pandas.DataFrame]: 
1030def filter_existing(
1031    self,
1032    df: 'pd.DataFrame',
1033    safe_copy: bool = True,
1034    date_bound_only: bool = False,
1035    include_unchanged_columns: bool = False,
1036    enforce_dtypes: bool = False,
1037    chunksize: Optional[int] = -1,
1038    debug: bool = False,
1039    as_polars: Optional[bool] = None,
1040    as_pandas: Optional[bool] = None,
1041    **kw
1042) -> Tuple['pd.DataFrame', 'pd.DataFrame', 'pd.DataFrame']:
1043    """
1044    Inspect a dataframe and filter out rows which already exist in the pipe.
1045
1046    Parameters
1047    ----------
1048    df: 'pd.DataFrame'
1049        The dataframe to inspect and filter.
1050        A `polars.DataFrame` or `LazyFrame` is accepted,
1051        in which case the returned dataframes are `polars.DataFrame`.
1052        Documents, dictionaries and Arrow data come back in the library of the
1053        `meerschaum:dataframe` setting.
1054
1055    safe_copy: bool, default True
1056        If `True`, create a copy before comparing and modifying the dataframes.
1057        Setting to `False` may mutate the DataFrames.
1058        See `meerschaum.utils.dataframe.filter_unseen_df`.
1059
1060    date_bound_only: bool, default False
1061        If `True`, only use the datetime index to fetch the sample dataframe.
1062
1063    include_unchanged_columns: bool, default False
1064        If `True`, include the backtrack columns which haven't changed in the update dataframe.
1065        This is useful if you can't update individual keys.
1066
1067    enforce_dtypes: bool, default False
1068        If `True`, ensure the given and intermediate dataframes are enforced to the correct dtypes.
1069        Setting `enforce_dtypes=True` may impact performance.
1070
1071    chunksize: Optional[int], default -1
1072        The `chunksize` used when fetching existing data.
1073
1074    debug: bool, default False
1075        Verbosity toggle.
1076
1077    as_polars: Optional[bool], default None
1078        For input that is not a DataFrame: `True` returns Polars frames, `False` pandas ones.
1079
1080    as_pandas: Optional[bool], default None
1081        For input that is not a DataFrame: `True` returns pandas frames, `False` Polars ones.
1082
1083    Returns
1084    -------
1085    A tuple of three DataFrames: unseen, update, and delta, of the library `df` came in
1086    (Polars for a Polars DataFrame or LazyFrame, pandas for a pandas DataFrame).
1087    """
1088    from meerschaum import _core
1089    from meerschaum.core.Pipe._data import _rust_context
1090    from meerschaum.utils.dataframe import (
1091        output_library_for, ensure_dataframe_library, frame_library_of,
1092    )
1093    context = _rust_context(self, debug=debug)
1094    if df is not None and not is_accepted_data(df):
1095        raise unsupported_data_error(df, f"filter the existing rows of {self}")
1096    library = (
1097        frame_library_of(df)
1098        or output_library_for(df, as_polars=as_polars, as_pandas=as_pandas)
1099    )
1100    ensure_dataframe_library(library)
1101    return _core.pipe_filter_existing(
1102        self,
1103        context,
1104        df,
1105        date_bound_only=date_bound_only,
1106        include_unchanged_columns=include_unchanged_columns,
1107        enforce_dtypes=enforce_dtypes,
1108        as_polars=(library == 'polars'),
1109        debug=debug,
1110    )

Inspect a dataframe and filter out rows which already exist in the pipe.

Parameters
  • df ('pd.DataFrame'): The dataframe to inspect and filter. A polars.DataFrame or LazyFrame is accepted, in which case the returned dataframes are polars.DataFrame. Documents, dictionaries and Arrow data come back in the library of the meerschaum:dataframe setting.
  • safe_copy (bool, default True): If True, create a copy before comparing and modifying the dataframes. Setting to False may mutate the DataFrames. See meerschaum.utils.dataframe.filter_unseen_df.
  • date_bound_only (bool, default False): If True, only use the datetime index to fetch the sample dataframe.
  • include_unchanged_columns (bool, default False): If True, include the backtrack columns which haven't changed in the update dataframe. This is useful if you can't update individual keys.
  • enforce_dtypes (bool, default False): If True, ensure the given and intermediate dataframes are enforced to the correct dtypes. Setting enforce_dtypes=True may impact performance.
  • chunksize (Optional[int], default -1): The chunksize used when fetching existing data.
  • debug (bool, default False): Verbosity toggle.
  • as_polars (Optional[bool], default None): For input that is not a DataFrame: True returns Polars frames, False pandas ones.
  • as_pandas (Optional[bool], default None): For input that is not a DataFrame: True returns pandas frames, False Polars ones.
Returns

  • A tuple of three DataFrames (unseen, update, and delta, of the library df came in):

  • (Polars for a Polars DataFrame or LazyFrame, pandas for a pandas DataFrame).

def get_num_workers(self, workers: int | None = None) -> int: 
1135def get_num_workers(self, workers: Optional[int] = None) -> int:
1136    """
1137    Get the number of workers to use for concurrent syncs.
1138
1139    Parameters
1140    ----------
1141    The number of workers passed via `--workers`.
1142
1143    Returns
1144    -------
1145    The number of workers, capped for safety.
1146    """
1147    is_thread_safe = getattr(self.instance_connector, 'IS_THREAD_SAFE', False)
1148    if not is_thread_safe:
1149        return 1
1150
1151    engine_pool_size = None
1152    current_num_connections = threading.active_count()
1153    if self.instance_connector.type == 'sql':
1154        ### The Rust driver owns the pool, sized by the flavor's `pool_size`. An SQLAlchemy engine
1155        ### (the `sql` extra) is consulted only when the caller already created one: building it
1156        ### here imported (and auto-installed) SQLAlchemy and the flavor's drivers on the first
1157        ### sync of every plugin pipe on a SQL instance.
1158        engine = self.instance_connector.__dict__.get('_engine', None)
1159        if engine is None:
1160            try:
1161                engine_pool_size = int(
1162                    self.instance_connector._sys_config.get('create_engine', {}).get('pool_size')
1163                    or (os.cpu_count() or 5)
1164                )
1165            except (TypeError, ValueError):
1166                engine_pool_size = os.cpu_count() or 5
1167        if engine is not None:
1168            engine_pool_size = engine.pool.size()
1169            current_num_connections = engine.pool.checkedout()
1170    desired_workers = (
1171        min(workers or engine_pool_size, engine_pool_size)
1172        if engine_pool_size is not None
1173        else workers
1174    )
1175    if desired_workers is None:
1176        desired_workers = (multiprocessing.cpu_count() if is_thread_safe else 1)
1177
1178    return max(
1179        (desired_workers - current_num_connections),
1180        1,
1181    )

Get the number of workers to use for concurrent syncs.

Parameters
  • The number of workers passed via --workers.
Returns
  • The number of workers, capped for safety.
def verify( self, begin: datetime.datetime | int | None = None, end: datetime.datetime | int | None = None, params: Dict[str, Any] | None = None, chunk_interval: datetime.timedelta | int | None = None, bounded: bool | None = None, deduplicate: bool = False, workers: int | None = None, batchsize: int | None = None, skip_chunks_with_greater_rowcounts: bool = False, check_rowcounts_only: bool = False, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
 40def verify(
 41    self,
 42    begin: Union[datetime, int, None] = None,
 43    end: Union[datetime, int, None] = None,
 44    params: Optional[Dict[str, Any]] = None,
 45    chunk_interval: Union[timedelta, int, None] = None,
 46    bounded: Optional[bool] = None,
 47    deduplicate: bool = False,
 48    workers: Optional[int] = None,
 49    batchsize: Optional[int] = None,
 50    skip_chunks_with_greater_rowcounts: bool = False,
 51    check_rowcounts_only: bool = False,
 52    debug: bool = False,
 53    **kwargs: Any
 54) -> SuccessTuple:
 55    """
 56    Verify the contents of the pipe by resyncing its interval.
 57
 58    Parameters
 59    ----------
 60    begin: Union[datetime, int, None], default None
 61        If specified, only verify rows greater than or equal to this value.
 62
 63    end: Union[datetime, int, None], default None
 64        If specified, only verify rows less than this value.
 65
 66    chunk_interval: Union[timedelta, int, None], default None
 67        If provided, use this as the size of the chunk boundaries.
 68        Default to the value set in `pipe.parameters['verify']['chunk_minutes']` (43200 — 30 days).
 69
 70    bounded: Optional[bool], default None
 71        If `True`, do not verify older than the oldest sync time or newer than the newest.
 72        If `False`, verify unbounded syncs outside of the new and old sync times.
 73        The default behavior (`None`) is to bound only if a bound interval is set
 74        (e.g. `pipe.parameters['verify']['bound_days']`).
 75
 76    deduplicate: bool, default False
 77        If `True`, deduplicate the pipe's table after the verification syncs.
 78
 79    workers: Optional[int], default None
 80        If provided, limit the verification to this many threads.
 81        Use a value of `1` to sync chunks in series.
 82
 83    batchsize: Optional[int], default None
 84        If provided, sync this many chunks in parallel.
 85        Defaults to `Pipe.get_num_workers()`.
 86
 87    skip_chunks_with_greater_rowcounts: bool, default False
 88        If `True`, compare the rowcounts for a chunk and skip syncing if the pipe's
 89        chunk rowcount equals or exceeds the remote's rowcount.
 90
 91    check_rowcounts_only: bool, default False
 92        If `True`, only compare rowcounts and print chunks which are out-of-sync.
 93
 94    debug: bool, default False
 95        Verbosity toggle.
 96
 97    kwargs: Any
 98        All keyword arguments are passed to `pipe.sync()`.
 99
100    Returns
101    -------
102    A SuccessTuple indicating whether the pipe was successfully resynced.
103    """
104    from meerschaum import _core
105    ### Verifying re-syncs its chunks: a `sync` lock refuses it before any is read.
106    if not check_rowcounts_only:
107        refusal = self._lock_refusal('sync', always=True, debug=debug)
108        if refusal:
109            return False, refusal
110    context, reason = _rust_verify_context(self, debug=debug)
111    if context is None:
112        return False, f"Cannot verify {self}: {reason}."
113    result = _core.verify_pipe(
114        self,
115        context['connectors'],
116        begin=begin,
117        end=end,
118        params=params,
119        chunk_interval=chunk_interval,
120        bounded=bounded,
121        deduplicate=deduplicate,
122        workers=workers,
123        batchsize=batchsize,
124        skip_chunks_with_greater_rowcounts=skip_chunks_with_greater_rowcounts,
125        check_rowcounts_only=check_rowcounts_only,
126        chunksize=kwargs.get('chunksize', -1),
127        check_existing=kwargs.get('check_existing', True),
128        kwargs={
129            key: value
130            for key, value in kwargs.items()
131            if key not in _VERIFY_SYNC_KWARGS
132        },
133        attributes=context['attributes'],
134        debug=debug,
135    )
136    ### The chunk syncs ran in Rust: drop what this object cached about the table.
137    self._invalidate_cache(hard=True, debug=debug)
138    self._clear_cache_key('_attributes_sync_time', debug=debug)
139    return result

Verify the contents of the pipe by resyncing its interval.

Parameters
  • begin (Union[datetime, int, None], default None): If specified, only verify rows greater than or equal to this value.
  • end (Union[datetime, int, None], default None): If specified, only verify rows less than this value.
  • chunk_interval (Union[timedelta, int, None], default None): If provided, use this as the size of the chunk boundaries. Default to the value set in pipe.parameters['verify']['chunk_minutes'] (43200, 30 days).
  • bounded (Optional[bool], default None): If True, do not verify older than the oldest sync time or newer than the newest. If False, verify unbounded syncs outside of the new and old sync times. The default behavior (None) is to bound only if a bound interval is set (e.g. pipe.parameters['verify']['bound_days']).
  • deduplicate (bool, default False): If True, deduplicate the pipe's table after the verification syncs.
  • workers (Optional[int], default None): If provided, limit the verification to this many threads. Use a value of 1 to sync chunks in series.
  • batchsize (Optional[int], default None): If provided, sync this many chunks in parallel. Defaults to Pipe.get_num_workers().
  • skip_chunks_with_greater_rowcounts (bool, default False): If True, compare the rowcounts for a chunk and skip syncing if the pipe's chunk rowcount equals or exceeds the remote's rowcount.
  • check_rowcounts_only (bool, default False): If True, only compare rowcounts and print chunks which are out-of-sync.
  • debug (bool, default False): Verbosity toggle.
  • kwargs (Any): All keyword arguments are passed to pipe.sync().
Returns
  • A SuccessTuple indicating whether the pipe was successfully resynced.
def verify_schema( self, apply: bool = False, dry_run: bool = False, debug: bool = False) -> Tuple[bool, str]: 
236def verify_schema(
237    self,
238    apply: bool = False,
239    dry_run: bool = False,
240    debug: bool = False,
241) -> SuccessTuple:
242    """
243    Compare the pipe's table with its declared dtypes (`dtypes`, the `columns` roles, geometry
244    notations) and, with `apply=True`, bring the table in line.
245
246    Added columns and safe widenings (`int` to `bigint`, `int` to `float`, `float` to `numeric`,
247    `varchar(n)` to `text`, `text` to `json`) keep every value; other type changes may fail on
248    existing values or change them. Columns are never dropped, and static pipes are included.
249    The command line equivalent is `mrsm verify schema`.
250
251    Parameters
252    ----------
253    apply: bool, default False
254        If `True`, apply every change the plan lists. Otherwise nothing is written.
255
256    dry_run: bool, default False
257        If `True`, return the SQL the plan would run (and write nothing).
258
259    debug: bool, default False
260        Verbosity toggle.
261
262    Returns
263    -------
264    A `SuccessTuple`: the message is the plan (a table of the columns that change), the SQL
265    with `dry_run=True`, or the plan and what changed with `apply=True`. A protected instance
266    refuses `apply=True` unless writes are allowed (`meerschaum.connectors.allow_protected()`).
267
268    Examples
269    --------
270    >>> pipe = mrsm.Pipe('plugin:noaa', 'weather', dtypes={'temperature': 'float'})
271    >>> success, plan = pipe.verify_schema()
272    >>> print(plan)
273    >>> success, msg = pipe.verify_schema(apply=True)
274    """
275    from meerschaum import _core
276    from meerschaum.core.Pipe._data import _rust_context
277    context = _rust_context(self, debug=debug)
278    success, message = _core.pipe_verify_schema(
279        self,
280        context,
281        apply=(apply and not dry_run),
282        dry_run=dry_run,
283        debug=debug,
284    )
285    if apply and not dry_run:
286        ### The table's columns changed: forget the cached columns types.
287        self._clear_cache_key('_columns_types', debug=debug)
288        self._clear_cache_key('_columns_types_timestamp', debug=debug)
289    return success, message

Compare the pipe's table with its declared dtypes (dtypes, the columns roles, geometry notations) and, with apply=True, bring the table in line.

Added columns and safe widenings (int to bigint, int to float, float to numeric, varchar(n) to text, text to json) keep every value; other type changes may fail on existing values or change them. Columns are never dropped, and static pipes are included. The command line equivalent is mrsm verify schema.

Parameters
  • apply (bool, default False): If True, apply every change the plan lists. Otherwise nothing is written.
  • dry_run (bool, default False): If True, return the SQL the plan would run (and write nothing).
  • debug (bool, default False): Verbosity toggle.
Returns

  • A SuccessTuple (the message is the plan (a table of the columns that change), the SQL):

  • with dry_run=True, or the plan and what changed with apply=True. A protected instance

  • refuses apply=True unless writes are allowed (meerschaum.connectors.allow_protected()).
Examples
>>> pipe = mrsm.Pipe('plugin:noaa', 'weather', dtypes={'temperature': 'float'})
>>> success, plan = pipe.verify_schema()
>>> print(plan)
>>> success, msg = pipe.verify_schema(apply=True)
def get_bound_interval(self, debug: bool = False) -> datetime.timedelta | int | None: 
143def get_bound_interval(self, debug: bool = False) -> Union[timedelta, int, None]:
144    """
145    Return the interval used to determine the bound time (limit for verification syncs).
146    If the datetime axis is an integer, just return its value.
147
148    Below are the supported keys for the bound interval:
149
150        - `pipe.parameters['verify']['bound_minutes']`
151        - `pipe.parameters['verify']['bound_hours']`
152        - `pipe.parameters['verify']['bound_days']`
153        - `pipe.parameters['verify']['bound_weeks']`
154        - `pipe.parameters['verify']['bound_years']`
155        - `pipe.parameters['verify']['bound_seconds']`
156
157    If multiple keys are present, the first on this priority list will be used.
158
159    Returns
160    -------
161    A `timedelta` or `int` value to be used to determine the bound time.
162    """
163    verify_params = self.parameters.get('verify', {})
164    prefix = 'bound_'
165    suffixes_to_check = ('minutes', 'hours', 'days', 'weeks', 'years', 'seconds')
166    keys_to_search = {
167        key: val
168        for key, val in verify_params.items()
169        if key.startswith(prefix)
170    }
171    bound_time_key, bound_time_value = None, None
172    for key, value in keys_to_search.items():
173        for suffix in suffixes_to_check:
174            if key == prefix + suffix:
175                bound_time_key = key
176                bound_time_value = value
177                break
178        if bound_time_key is not None:
179            break
180
181    if bound_time_value is None:
182        return bound_time_value
183
184    dt_col = self.columns.get('datetime', None)
185    if not dt_col:
186        return bound_time_value
187
188    dt_typ = self.dtypes.get(dt_col, 'datetime')
189    if 'int' in dt_typ.lower():
190        return int(bound_time_value)
191
192    interval_type = bound_time_key.replace(prefix, '')
193    return timedelta(**{interval_type: bound_time_value})

Return the interval used to determine the bound time (limit for verification syncs). If the datetime axis is an integer, just return its value.

Below are the supported keys for the bound interval:

- `pipe.parameters['verify']['bound_minutes']`
- `pipe.parameters['verify']['bound_hours']`
- `pipe.parameters['verify']['bound_days']`
- `pipe.parameters['verify']['bound_weeks']`
- `pipe.parameters['verify']['bound_years']`
- `pipe.parameters['verify']['bound_seconds']`

If multiple keys are present, the first on this priority list will be used.

Returns
  • A timedelta or int value to be used to determine the bound time.
def get_bound_time(self, debug: bool = False) -> datetime.datetime | int | None: 
196def get_bound_time(self, debug: bool = False) -> Union[datetime, int, None]:
197    """
198    The bound time is the limit at which long-running verification syncs should stop.
199    A value of `None` means verification syncs should be unbounded.
200
201    Like deriving a backtrack time from `pipe.get_sync_time()`,
202    the bound time is the sync time minus a large window (e.g. 366 days).
203
204    Unbound verification syncs (i.e. `bound_time is None`)
205    if the oldest sync time is less than the bound interval.
206
207    Returns
208    -------
209    A `datetime` or `int` corresponding to the
210    `begin` bound for verification and deduplication syncs.
211    """
212    bound_interval = self.get_bound_interval(debug=debug)
213    if bound_interval is None:
214        return None
215
216    sync_time = self.get_sync_time(debug=debug)
217    if sync_time is None:
218        return None
219
220    bound_time = sync_time - bound_interval
221    oldest_sync_time = self.get_sync_time(newest=False, debug=debug)
222    max_bound_time_days = STATIC_CONFIG['pipes']['max_bound_time_days']
223
224    extreme_sync_times_delta = (
225        hasattr(oldest_sync_time, 'tzinfo')
226        and (sync_time - oldest_sync_time) >= timedelta(days=max_bound_time_days)
227    )
228
229    return (
230        bound_time
231        if bound_time > oldest_sync_time or extreme_sync_times_delta
232        else None
233    )

The bound time is the limit at which long-running verification syncs should stop. A value of None means verification syncs should be unbounded.

Like deriving a backtrack time from pipe.get_sync_time(), the bound time is the sync time minus a large window (e.g. 366 days).

Unbound verification syncs (i.e. bound_time is None) if the oldest sync time is less than the bound interval.

Returns
  • A datetime or int corresponding to the
  • begin bound for verification and deduplication syncs.
def delete(self, drop: bool = True, debug: bool = False, **kw) -> Tuple[bool, str]: 
12def delete(
13    self,
14    drop: bool = True,
15    debug: bool = False,
16    **kw
17) -> SuccessTuple:
18    """
19    Call the Pipe's instance connector's `delete_pipe()` method.
20
21    Parameters
22    ----------
23    drop: bool, default True
24        If `True`, drop the pipes' target table.
25
26    debug : bool, default False
27        Verbosity toggle.
28
29    Returns
30    -------
31    A `SuccessTuple` of success (`bool`), message (`str`).
32
33    """
34    from meerschaum.utils.warnings import warn
35    from meerschaum.utils.venv import Venv
36    from meerschaum.connectors import get_connector_plugin
37
38    if self.temporary:
39        if self.cache:
40            invalidate_success, invalidate_msg = self._invalidate_cache(hard=True, debug=debug)
41            if not invalidate_success:
42                return invalidate_success, invalidate_msg
43
44        return (
45            False,
46            "Cannot delete pipes created with `temporary=True` (read-only). "
47            + "You may want to call `pipe.drop()` instead."
48        )
49
50    ### Deleting drops the table first: a `drop` or `delete` lock refuses it before anything
51    ### changes, whatever the instance.
52    refusal = self._lock_refusal('delete_and_drop' if drop else 'delete', always=True, debug=debug)
53    if refusal:
54        return False, refusal
55
56    if drop:
57        drop_success, drop_msg = self.drop(debug=debug)
58        if not drop_success:
59            warn(f"Failed to drop {self}:\n{drop_msg}")
60
61    with Venv(get_connector_plugin(self.instance_connector)):
62        result = self.instance_connector.delete_pipe(self, debug=debug, **kw)
63
64    if not isinstance(result, tuple):
65        return False, f"Received an unexpected result from '{self.instance_connector}': {result}"
66
67    if result[0]:
68        self._invalidate_cache(hard=True, debug=debug)
69        self._clear_cache_key('_id', debug=debug)
70        from meerschaum.core.Pipeline._engine import forget_engine_state
71        forget_engine_state(self, 'delete_pipe')
72
73    return result

Call the Pipe's instance connector's delete_pipe() method.

Parameters
  • drop (bool, default True): If True, drop the pipes' target table.
  • debug (bool, default False): Verbosity toggle.
Returns
def drop(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
14def drop(
15    self,
16    debug: bool = False,
17    **kw: Any
18) -> SuccessTuple:
19    """
20    Call the Pipe's instance connector's `drop_pipe()` method.
21
22    Parameters
23    ----------
24    debug: bool, default False:
25        Verbosity toggle.
26
27    Returns
28    -------
29    A `SuccessTuple` of success, message.
30
31    """
32    from meerschaum.utils.venv import Venv
33    from meerschaum.connectors import get_connector_plugin
34
35    ### A `drop` lock refuses it (the core checks its own instances).
36    refusal = self._lock_refusal('drop', debug=debug)
37    if refusal:
38        return False, refusal
39
40    self._clear_cache_key('_exists', debug=debug)
41
42    with Venv(get_connector_plugin(self.instance_connector)):
43        if hasattr(self.instance_connector, 'drop_pipe'):
44            result = self.instance_connector.drop_pipe(self, debug=debug, **kw)
45        else:
46            result = (
47                False,
48                (
49                    "Cannot drop pipes for instance connectors of type "
50                    f"'{self.instance_connector.type}'."
51                )
52            )
53
54    self._clear_cache_key('_exists', debug=debug)
55    self._clear_cache_key('_exists_timestamp', debug=debug)
56
57    if isinstance(result, tuple) and result and result[0]:
58        from meerschaum.core.Pipeline._engine import forget_engine_state
59        forget_engine_state(self, 'drop_pipe')
60
61    return result

Call the Pipe's instance connector's drop_pipe() method.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
def drop_indices( self, columns: List[str] | None = None, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
 64def drop_indices(
 65    self,
 66    columns: Optional[List[str]] = None,
 67    debug: bool = False,
 68    **kw: Any
 69) -> SuccessTuple:
 70    """
 71    Call the Pipe's instance connector's `drop_indices()` method.
 72
 73    Parameters
 74    ----------
 75    columns: Optional[List[str]] = None
 76        If provided, only drop indices in the given list.
 77
 78    debug: bool, default False:
 79        Verbosity toggle.
 80
 81    Returns
 82    -------
 83    A `SuccessTuple` of success, message.
 84
 85    """
 86    from meerschaum.utils.venv import Venv
 87    from meerschaum.connectors import get_connector_plugin
 88
 89    ### Dropping indices changes the table's schema: a `drop` lock refuses it.
 90    refusal = self._lock_refusal('schema', debug=debug)
 91    if refusal:
 92        return False, refusal
 93
 94    self._clear_cache_key('_columns_indices', debug=debug)
 95    self._clear_cache_key('_columns_indices_timestamp', debug=debug)
 96    self._clear_cache_key('_columns_types', debug=debug)
 97    self._clear_cache_key('_columns_types_timestamp', debug=debug)
 98
 99    with Venv(get_connector_plugin(self.instance_connector)):
100        if hasattr(self.instance_connector, 'drop_pipe_indices'):
101            result = self.instance_connector.drop_pipe_indices(
102                self,
103                columns=columns,
104                debug=debug,
105                **kw
106            )
107        else:
108            result = (
109                False,
110                (
111                    "Cannot drop indices for instance connectors of type "
112                    f"'{self.instance_connector.type}'."
113                )
114            )
115
116    self._clear_cache_key('_columns_indices', debug=debug)
117    self._clear_cache_key('_columns_indices_timestamp', debug=debug)
118    self._clear_cache_key('_columns_types', debug=debug)
119    self._clear_cache_key('_columns_types_timestamp', debug=debug)
120
121    return result

Call the Pipe's instance connector's drop_indices() method.

Parameters
  • columns (Optional[List[str]] = None): If provided, only drop indices in the given list.
  • debug (bool, default False:): Verbosity toggle.
Returns
def compress(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
14def compress(
15    self,
16    debug: bool = False,
17    **kw: Any
18) -> SuccessTuple:
19    """
20    Call the Pipe's instance connector's `compress_pipe()` method.
21
22    For TimescaleDB hypertables this enables and applies native compression.
23    Other flavors fall back to their respective compression mechanisms where supported.
24
25    Parameters
26    ----------
27    debug: bool, default False:
28        Verbosity toggle.
29
30    Returns
31    -------
32    A `SuccessTuple` of success, message.
33    """
34    from meerschaum.utils.venv import Venv
35    from meerschaum.connectors import get_connector_plugin
36
37    try:
38        with Venv(get_connector_plugin(self.instance_connector)):
39            if hasattr(self.instance_connector, 'compress_pipe'):
40                result = self.instance_connector.compress_pipe(self, debug=debug, **kw)
41            else:
42                result = (
43                    False,
44                    (
45                        "Cannot compress pipes for instance connectors of type "
46                        f"'{self.instance_connector.type}'."
47                    )
48                )
49    except NotImplementedError:
50        result = (
51            False,
52            (
53                "Compression is not implemented for instance connectors of type "
54                f"'{self.instance_connector.type}'."
55            )
56        )
57
58    self._clear_cache_key('_exists', debug=debug)
59    return result

Call the Pipe's instance connector's compress_pipe() method.

For TimescaleDB hypertables this enables and applies native compression. Other flavors fall back to their respective compression mechanisms where supported.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
def decompress(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
 62def decompress(
 63    self,
 64    debug: bool = False,
 65    **kw: Any
 66) -> SuccessTuple:
 67    """
 68    Call the Pipe's instance connector's `decompress_pipe()` method, the inverse of `compress()`.
 69
 70    For TimescaleDB hypertables this removes the compression policy and converts compressed
 71    chunks back to row-store. Other flavors fall back to their respective mechanisms where
 72    supported.
 73
 74    Parameters
 75    ----------
 76    debug: bool, default False:
 77        Verbosity toggle.
 78
 79    Returns
 80    -------
 81    A `SuccessTuple` of success, message.
 82    """
 83    from meerschaum.utils.venv import Venv
 84    from meerschaum.connectors import get_connector_plugin
 85
 86    try:
 87        with Venv(get_connector_plugin(self.instance_connector)):
 88            if hasattr(self.instance_connector, 'decompress_pipe'):
 89                result = self.instance_connector.decompress_pipe(self, debug=debug, **kw)
 90            else:
 91                result = (
 92                    False,
 93                    (
 94                        "Cannot decompress pipes for instance connectors of type "
 95                        f"'{self.instance_connector.type}'."
 96                    )
 97                )
 98    except NotImplementedError:
 99        result = (
100            False,
101            (
102                "Decompression is not implemented for instance connectors of type "
103                f"'{self.instance_connector.type}'."
104            )
105        )
106
107    self._clear_cache_key('_exists', debug=debug)
108    return result

Call the Pipe's instance connector's decompress_pipe() method, the inverse of compress().

For TimescaleDB hypertables this removes the compression policy and converts compressed chunks back to row-store. Other flavors fall back to their respective mechanisms where supported.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
def vacuum( self, full: bool = False, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
14def vacuum(
15    self,
16    full: bool = False,
17    debug: bool = False,
18    **kw: Any
19) -> SuccessTuple:
20    """
21    Call the Pipe's instance connector's `vacuum_pipe()` method to reclaim disk space.
22
23    For PostgreSQL-family tables this runs `VACUUM` (optionally `VACUUM FULL`); other flavors
24    fall back to their respective space-reclaiming mechanisms where supported.
25
26    Parameters
27    ----------
28    full: bool, default False
29        If `True` (PostgreSQL family only), run `VACUUM FULL` to return freed space to the OS.
30
31    debug: bool, default False
32        Verbosity toggle.
33
34    Returns
35    -------
36    A `SuccessTuple` of success, message.
37    """
38    from meerschaum.utils.venv import Venv
39    from meerschaum.connectors import get_connector_plugin
40
41    try:
42        with Venv(get_connector_plugin(self.instance_connector)):
43            if hasattr(self.instance_connector, 'vacuum_pipe'):
44                result = self.instance_connector.vacuum_pipe(self, full=full, debug=debug, **kw)
45            else:
46                result = (
47                    False,
48                    (
49                        "Cannot vacuum pipes for instance connectors of type "
50                        f"'{self.instance_connector.type}'."
51                    )
52                )
53    except NotImplementedError:
54        result = (
55            False,
56            (
57                "Vacuuming is not implemented for instance connectors of type "
58                f"'{self.instance_connector.type}'."
59            )
60        )
61
62    self._clear_cache_key('_exists', debug=debug)
63    return result

Call the Pipe's instance connector's vacuum_pipe() method to reclaim disk space.

For PostgreSQL-family tables this runs VACUUM (optionally VACUUM FULL); other flavors fall back to their respective space-reclaiming mechanisms where supported.

Parameters
  • full (bool, default False): If True (PostgreSQL family only), run VACUUM FULL to return freed space to the OS.
  • debug (bool, default False): Verbosity toggle.
Returns
def analyze(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
 66def analyze(
 67    self,
 68    debug: bool = False,
 69    **kw: Any
 70) -> SuccessTuple:
 71    """
 72    Call the Pipe's instance connector's `analyze_pipe()` method to refresh planner statistics.
 73
 74    Parameters
 75    ----------
 76    debug: bool, default False
 77        Verbosity toggle.
 78
 79    Returns
 80    -------
 81    A `SuccessTuple` of success, message.
 82    """
 83    from meerschaum.utils.venv import Venv
 84    from meerschaum.connectors import get_connector_plugin
 85
 86    try:
 87        with Venv(get_connector_plugin(self.instance_connector)):
 88            if hasattr(self.instance_connector, 'analyze_pipe'):
 89                result = self.instance_connector.analyze_pipe(self, debug=debug, **kw)
 90            else:
 91                result = (
 92                    False,
 93                    (
 94                        "Cannot analyze pipes for instance connectors of type "
 95                        f"'{self.instance_connector.type}'."
 96                    )
 97                )
 98    except NotImplementedError:
 99        result = (
100            False,
101            (
102                "Analyzing is not implemented for instance connectors of type "
103                f"'{self.instance_connector.type}'."
104            )
105        )
106
107    return result

Call the Pipe's instance connector's analyze_pipe() method to refresh planner statistics.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
def repartition( self, chunk_minutes: int | None = None, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
110def repartition(
111    self,
112    chunk_minutes: Optional[int] = None,
113    debug: bool = False,
114    **kw: Any
115) -> SuccessTuple:
116    """
117    Call the Pipe's instance connector's `partition_pipe()` method to rebuild the target table
118    to a new partition (chunk) width.
119
120    On TimescaleDB this changes the chunk interval for future chunks. On PostgreSQL / PostGIS,
121    MySQL / MariaDB, and MSSQL it rebuilds the natively range-partitioned table at the new width.
122
123    Parameters
124    ----------
125    chunk_minutes: Optional[int], default None
126        The new partition width in minutes. Defaults to the pipe's `verify.chunk_minutes`.
127
128    debug: bool, default False
129        Verbosity toggle.
130
131    Returns
132    -------
133    A `SuccessTuple` of success, message.
134    """
135    from meerschaum.utils.venv import Venv
136    from meerschaum.connectors import get_connector_plugin
137
138    ### Repartitioning rebuilds the table (`drop`) and records its width (`parameters`).
139    for op in ('schema', 'edit'):
140        refusal = self._lock_refusal(op, debug=debug)
141        if refusal:
142            return False, refusal
143
144    try:
145        with Venv(get_connector_plugin(self.instance_connector)):
146            if hasattr(self.instance_connector, 'partition_pipe'):
147                result = self.instance_connector.partition_pipe(
148                    self, chunk_minutes=chunk_minutes, debug=debug, **kw
149                )
150            else:
151                result = (
152                    False,
153                    (
154                        "Cannot repartition pipes for instance connectors of type "
155                        f"'{self.instance_connector.type}'."
156                    )
157                )
158    except NotImplementedError:
159        result = (
160            False,
161            (
162                "Repartitioning is not implemented for instance connectors of type "
163                f"'{self.instance_connector.type}'."
164            )
165        )
166
167    self._clear_cache_key('_exists', debug=debug)
168    return result

Call the Pipe's instance connector's partition_pipe() method to rebuild the target table to a new partition (chunk) width.

On TimescaleDB this changes the chunk interval for future chunks. On PostgreSQL / PostGIS, MySQL / MariaDB, and MSSQL it rebuilds the natively range-partitioned table at the new width.

Parameters
  • chunk_minutes (Optional[int], default None): The new partition width in minutes. Defaults to the pipe's verify.chunk_minutes.
  • debug (bool, default False): Verbosity toggle.
Returns
def create_indices( self, columns: List[str] | None = None, debug: bool = False, **kw: Any) -> Tuple[bool, str]: 
14def create_indices(
15    self,
16    columns: Optional[List[str]] = None,
17    debug: bool = False,
18    **kw: Any
19) -> SuccessTuple:
20    """
21    Call the Pipe's instance connector's `create_pipe_indices()` method.
22
23    Parameters
24    ----------
25    debug: bool, default False:
26        Verbosity toggle.
27
28    Returns
29    -------
30    A `SuccessTuple` of success, message.
31
32    """
33    from meerschaum.utils.venv import Venv
34    from meerschaum.connectors import get_connector_plugin
35
36    self._clear_cache_key('_columns_indices', debug=debug)
37    self._clear_cache_key('_columns_indices_timestamp', debug=debug)
38    self._clear_cache_key('_columns_types', debug=debug)
39    self._clear_cache_key('_columns_types_timestamp', debug=debug)
40
41    with Venv(get_connector_plugin(self.instance_connector)):
42        if hasattr(self.instance_connector, 'create_pipe_indices'):
43            result = self.instance_connector.create_pipe_indices(
44                self,
45                columns=columns,
46                debug=debug,
47                **kw
48            )
49        else:
50            result = (
51                False,
52                (
53                    "Cannot create indices for instance connectors of type "
54                    f"'{self.instance_connector.type}'."
55                )
56            )
57
58    self._clear_cache_key('_columns_indices', debug=debug)
59    self._clear_cache_key('_columns_indices_timestamp', debug=debug)
60    self._clear_cache_key('_columns_types', debug=debug)
61    self._clear_cache_key('_columns_types_timestamp', debug=debug)
62
63    return result

Call the Pipe's instance connector's create_pipe_indices() method.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
def clear( self, begin: datetime.datetime | None = None, end: datetime.datetime | None = None, params: Dict[str, Any] | None = None, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
16def clear(
17    self,
18    begin: Optional[datetime] = None,
19    end: Optional[datetime] = None,
20    params: Optional[Dict[str, Any]] = None,
21    debug: bool = False,
22    **kwargs: Any
23) -> SuccessTuple:
24    """
25    Call the Pipe's instance connector's `clear_pipe` method.
26
27    Parameters
28    ----------
29    begin: Optional[datetime], default None:
30        If provided, only remove rows newer than this datetime value.
31
32    end: Optional[datetime], default None:
33        If provided, only remove rows older than this datetime column (not including end).
34
35    params: Optional[Dict[str, Any]], default None
36         See `meerschaum.utils.sql.build_where`.
37
38    debug: bool, default False:
39        Verbositity toggle.
40
41    Returns
42    -------
43    A `SuccessTuple` corresponding to whether this procedure completed successfully.
44
45    Examples
46    --------
47    >>> pipe = mrsm.Pipe('test', 'test', columns={'datetime': 'dt'}, instance='sql:local')
48    >>> pipe.sync({'dt': [datetime(2020, 1, 1, 0, 0)]})
49    >>> pipe.sync({'dt': [datetime(2021, 1, 1, 0, 0)]})
50    >>> pipe.sync({'dt': [datetime(2022, 1, 1, 0, 0)]})
51    >>> 
52    >>> pipe.clear(begin=datetime(2021, 1, 1, 0, 0))
53    >>> pipe.get_data()
54              dt
55    0 2020-01-01
56
57    """
58    from meerschaum.utils.warnings import warn
59    from meerschaum.utils.venv import Venv
60    from meerschaum.connectors import get_connector_plugin
61
62    begin, end = self.parse_date_bounds(begin, end)
63
64    ### A `clear` lock refuses row deletes (the core checks its own instances).
65    refusal = self._lock_refusal('clear', debug=debug)
66    if refusal:
67        return False, refusal
68
69    with Venv(get_connector_plugin(self.instance_connector)):
70        result = self.instance_connector.clear_pipe(
71            self,
72            begin=begin,
73            end=end,
74            params=params,
75            debug=debug,
76            **kwargs
77        )
78    if begin is None and end is None and not params and isinstance(result, tuple) and result[0]:
79        from meerschaum.core.Pipeline._engine import forget_engine_state
80        forget_engine_state(self, 'clear_pipe')
81    return result

Call the Pipe's instance connector's clear_pipe method.

Parameters
  • begin (Optional[datetime], default None:): If provided, only remove rows newer than this datetime value.
  • end (Optional[datetime], default None:): If provided, only remove rows older than this datetime column (not including end).
  • params (Optional[Dict[str, Any]], default None): See meerschaum.utils.sql.build_where.
  • debug (bool, default False:): Verbositity toggle.
Returns
  • A SuccessTuple corresponding to whether this procedure completed successfully.
Examples
>>> pipe = mrsm.Pipe('test', 'test', columns={'datetime': 'dt'}, instance='sql:local')
>>> pipe.sync({'dt': [datetime(2020, 1, 1, 0, 0)]})
>>> pipe.sync({'dt': [datetime(2021, 1, 1, 0, 0)]})
>>> pipe.sync({'dt': [datetime(2022, 1, 1, 0, 0)]})
>>> 
>>> pipe.clear(begin=datetime(2021, 1, 1, 0, 0))
>>> pipe.get_data()
          dt
0 2020-01-01
def deduplicate( self, begin: datetime.datetime | int | None = None, end: datetime.datetime | int | None = None, params: Dict[str, Any] | None = None, chunk_interval: datetime.datetime | int | None = None, bounded: bool | None = None, workers: int | None = None, debug: bool = False, _use_instance_method: bool = True, **kwargs: Any) -> Tuple[bool, str]: 
15def deduplicate(
16    self,
17    begin: Union[datetime, int, None] = None,
18    end: Union[datetime, int, None] = None,
19    params: Optional[Dict[str, Any]] = None,
20    chunk_interval: Union[datetime, int, None] = None,
21    bounded: Optional[bool] = None,
22    workers: Optional[int] = None,
23    debug: bool = False,
24    _use_instance_method: bool = True,
25    **kwargs: Any
26) -> SuccessTuple:
27    """
28    Call the Pipe's instance connector's `delete_duplicates` method to delete duplicate rows.
29
30    Parameters
31    ----------
32    begin: Union[datetime, int, None], default None:
33        If provided, only deduplicate rows newer than this datetime value.
34
35    end: Union[datetime, int, None], default None:
36        If provided, only deduplicate rows older than this datetime column (not including end).
37
38    params: Optional[Dict[str, Any]], default None
39        Restrict deduplication to this filter (for multiplexed data streams).
40        See `meerschaum.utils.sql.build_where`.
41
42    chunk_interval: Union[timedelta, int, None], default None
43        If provided, use this for the chunk bounds.
44        Defaults to the value set in `pipe.parameters['verify']['chunk_minutes']` (43200 — 30 days).
45
46    bounded: Optional[bool], default None
47        Only check outside the oldest and newest sync times if bounded is explicitly `False`.
48
49    workers: Optional[int], default None
50        If the instance connector is thread-safe, limit concurrenct syncs to this many threads.
51
52    debug: bool, default False:
53        Verbositity toggle.
54
55    kwargs: Any
56        All other keyword arguments are passed to
57        `pipe.sync()`, `pipe.clear()`, and `pipe.get_data().
58
59    Returns
60    -------
61    A `SuccessTuple` corresponding to whether all of the chunks were successfully deduplicated.
62    """
63    from meerschaum import _core
64    from meerschaum.core.Pipe._data import _rust_context
65    context = _rust_context(self, debug=debug)
66    begin, end = self.parse_date_bounds(begin, end)
67    return _core.pipe_deduplicate(
68        self,
69        context,
70        begin=begin,
71        end=end,
72        params=params,
73        chunk_interval=chunk_interval,
74        bounded=bounded,
75        use_instance_method=_use_instance_method,
76        workers=self.get_num_workers(workers=workers),
77        kwargs=kwargs,
78        debug=debug,
79    )

Call the Pipe's instance connector's delete_duplicates method to delete duplicate rows.

Parameters
  • begin (Union[datetime, int, None], default None:): If provided, only deduplicate rows newer than this datetime value.
  • end (Union[datetime, int, None], default None:): If provided, only deduplicate rows older than this datetime column (not including end).
  • params (Optional[Dict[str, Any]], default None): Restrict deduplication to this filter (for multiplexed data streams). See meerschaum.utils.sql.build_where.
  • chunk_interval (Union[timedelta, int, None], default None): If provided, use this for the chunk bounds. Defaults to the value set in pipe.parameters['verify']['chunk_minutes'] (43200, 30 days).
  • bounded (Optional[bool], default None): Only check outside the oldest and newest sync times if bounded is explicitly False.
  • workers (Optional[int], default None): If the instance connector is thread-safe, limit concurrenct syncs to this many threads.
  • debug (bool, default False:): Verbositity toggle.
  • kwargs (Any): All other keyword arguments are passed to pipe.sync(), pipe.clear(), and `pipe.get_data().
Returns
  • A SuccessTuple corresponding to whether all of the chunks were successfully deduplicated.
def bootstrap( self, debug: bool = False, yes: bool = False, force: bool = False, noask: bool = False, shell: bool = False, **kw) -> Tuple[bool, str]: 
 16def bootstrap(
 17    self,
 18    debug: bool = False,
 19    yes: bool = False,
 20    force: bool = False,
 21    noask: bool = False,
 22    shell: bool = False,
 23    **kw
 24) -> SuccessTuple:
 25    """
 26    Prompt the user to create a pipe's requirements all from one method.
 27    This method shouldn't be used in any automated scripts because it interactively
 28    prompts the user and therefore may hang.
 29
 30    Parameters
 31    ----------
 32    debug: bool, default False:
 33        Verbosity toggle.
 34
 35    yes: bool, default False:
 36        Print the questions and automatically agree.
 37
 38    force: bool, default False:
 39        Skip the questions and agree anyway.
 40
 41    noask: bool, default False:
 42        Print the questions but go with the default answer.
 43
 44    shell: bool, default False:
 45        Used to determine if we are in the interactive shell.
 46        
 47    Returns
 48    -------
 49    A `SuccessTuple` corresponding to the success of this procedure.
 50
 51    """
 52
 53    from meerschaum.utils.warnings import info
 54    from meerschaum.utils.prompt import prompt, yes_no
 55    from meerschaum.utils.formatting import pprint
 56    from meerschaum.config import get_config
 57    from meerschaum.utils.formatting._shell import clear_screen
 58    from meerschaum.utils.formatting import print_tuple
 59    from meerschaum.actions import actions
 60    from meerschaum.utils.venv import Venv
 61    from meerschaum.connectors import get_connector_plugin
 62
 63    _clear = get_config('shell', 'clear_screen', patch=True)
 64
 65    if self.id is not None:
 66        delete_tuple = self.delete(debug=debug)
 67        if not delete_tuple[0]:
 68            return delete_tuple
 69
 70    if _clear:
 71        clear_screen(debug=debug)
 72
 73    _parameters = _get_parameters(self, debug=debug)
 74    self.parameters = _parameters
 75    pprint(self.parameters)
 76    try:
 77        prompt(
 78            f"\n    Press [Enter] to register {self} with the above configuration:",
 79            icon = False
 80        )
 81    except KeyboardInterrupt:
 82        return False, f"Aborted bootstrapping {self}."
 83
 84    with Venv(get_connector_plugin(self.instance_connector)):
 85        register_tuple = self.instance_connector.register_pipe(self, debug=debug)
 86
 87    if not register_tuple[0]:
 88        return register_tuple
 89
 90    if _clear:
 91        clear_screen(debug=debug)
 92
 93    try:
 94        if yes_no(
 95            f"Would you like to edit the definition for {self}?",
 96            yes=yes,
 97            noask=noask,
 98            default='n',
 99        ):
100            edit_tuple = self.edit_definition(debug=debug)
101            if not edit_tuple[0]:
102                return edit_tuple
103
104        if yes_no(
105            f"Would you like to try syncing {self} now?",
106            yes=yes,
107            noask=noask,
108            default='n',
109        ):
110            sync_tuple = actions['sync'](
111                ['pipes'],
112                connector_keys=[self.connector_keys],
113                metric_keys=[self.metric_key],
114                location_keys=[self.location_key],
115                mrsm_instance=str(self.instance_connector),
116                debug=debug,
117                shell=shell,
118            )
119            if not sync_tuple[0]:
120                return sync_tuple
121    except Exception as e:
122        return False, f"Failed to bootstrap {self}:\n" + str(e)
123
124    print_tuple((True, f"Finished bootstrapping {self}!"))
125    info(
126        "You can edit this pipe later with `edit pipes` "
127        + "or set the definition with `edit pipes definition`.\n"
128        + "    To sync data into your pipe, run `sync pipes`."
129    )
130
131    return True, "Success"

Prompt the user to create a pipe's requirements all from one method. This method shouldn't be used in any automated scripts because it interactively prompts the user and therefore may hang.

Parameters
  • debug (bool, default False:): Verbosity toggle.
  • yes (bool, default False:): Print the questions and automatically agree.
  • force (bool, default False:): Skip the questions and agree anyway.
  • noask (bool, default False:): Print the questions but go with the default answer.
  • shell (bool, default False:): Used to determine if we are in the interactive shell.
Returns
  • A SuccessTuple corresponding to the success of this procedure.
def enforce_dtypes( self, df: pandas.DataFrame, chunksize: int | None = -1, enforce: bool = True, safe_copy: bool = True, dtypes: Dict[str, str] | None = None, as_polars: bool | None = None, debug: bool = False, as_pandas: bool | None = None) -> pandas.DataFrame: 
19def enforce_dtypes(
20    self,
21    df: 'pd.DataFrame',
22    chunksize: Optional[int] = -1,
23    enforce: bool = True,
24    safe_copy: bool = True,
25    dtypes: Optional[Dict[str, str]] = None,
26    as_polars: Optional[bool] = None,
27    debug: bool = False,
28    as_pandas: Optional[bool] = None,
29) -> 'pd.DataFrame':
30    """
31    Cast the input dataframe to the pipe's registered data types.
32    If the pipe does not exist and dtypes are not set, return the dataframe.
33
34    The result is the library `df` came in (a Polars frame stays Polars, a pandas frame
35    pandas); documents, dictionaries and Arrow data come back in the library of the
36    `meerschaum:dataframe` setting. `as_polars=True` / `as_pandas=True` ask for one library.
37    """
38    from meerschaum import _core
39    from meerschaum.core.Pipe._data import _rust_context
40    from meerschaum.core.Pipe._sync import is_accepted_data, unsupported_data_error
41    from meerschaum.utils.dataframe import output_library_for, ensure_dataframe_library
42    context = _rust_context(self, debug=debug)
43    if df is not None and not is_accepted_data(df):
44        raise unsupported_data_error(df, f"enforce the dtypes of {self}")
45    library = output_library_for(df, as_polars=as_polars, as_pandas=as_pandas)
46    if df is not None:
47        ensure_dataframe_library(library)
48    as_polars = library == 'polars'
49    return _core.pipe_enforce_dtypes(
50        self,
51        context,
52        df,
53        enforce=enforce,
54        safe_copy=safe_copy,
55        dtypes=dtypes,
56        as_polars=as_polars,
57        debug=debug,
58    )

Cast the input dataframe to the pipe's registered data types. If the pipe does not exist and dtypes are not set, return the dataframe.

The result is the library df came in (a Polars frame stays Polars, a pandas frame pandas); documents, dictionaries and Arrow data come back in the library of the meerschaum:dataframe setting. as_polars=True / as_pandas=True ask for one library.

def infer_dtypes( self, persist: bool = False, refresh: bool = False, debug: bool = False) -> Dict[str, Any]: 
 61def infer_dtypes(
 62    self,
 63    persist: bool = False,
 64    refresh: bool = False,
 65    debug: bool = False,
 66) -> Dict[str, Any]:
 67    """
 68    If `dtypes` is not set in `meerschaum.Pipe.parameters`,
 69    infer the data types from the underlying table if it exists.
 70
 71    Parameters
 72    ----------
 73    persist: bool, default False
 74        If `True`, persist the inferred data types to `meerschaum.Pipe.parameters`.
 75        NOTE: Use with caution! Generally `dtypes` is meant to be user-configurable only.
 76
 77    refresh: bool, default False
 78        If `True`, retrieve the latest columns-types for the pipe.
 79        See `Pipe.get_columns.types()`.
 80
 81    Returns
 82    -------
 83    A dictionary of strings containing the pandas data types for this Pipe.
 84    """
 85    from meerschaum import _core
 86    from meerschaum.core.Pipe._data import _rust_context
 87    context = _rust_context(self, debug=debug, refresh=refresh)
 88    remote_pd_dtypes = _core.pipe_infer_dtypes(self, context, refresh=refresh, debug=debug)
 89    ### A table that does not exist has no dtypes to persist.
 90    if not persist or not remote_pd_dtypes:
 91        return remote_pd_dtypes
 92    parameters = self.get_parameters(refresh=refresh, debug=debug)
 93    dtypes = parameters.get('dtypes', {})
 94    dtypes.update({
 95        col: typ
 96        for col, typ in remote_pd_dtypes.items()
 97        if col not in dtypes
 98    })
 99    self.dtypes = dtypes
100    self.edit(interactive=False, debug=debug)
101    return remote_pd_dtypes

If dtypes is not set in meerschaum.Pipe.parameters, infer the data types from the underlying table if it exists.

Parameters
  • persist (bool, default False): If True, persist the inferred data types to meerschaum.Pipe.parameters. NOTE: Use with caution! Generally dtypes is meant to be user-configurable only.
  • refresh (bool, default False): If True, retrieve the latest columns-types for the pipe. See Pipe.get_columns.types().
Returns
  • A dictionary of strings containing the pandas data types for this Pipe.
def copy_to( self, instance_keys: str, sync: bool = True, begin: datetime.datetime | int | None = None, end: datetime.datetime | int | None = None, params: Dict[str, Any] | None = None, chunk_interval: datetime.timedelta | int | None = None, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
 15def copy_to(
 16    self,
 17    instance_keys: str,
 18    sync: bool = True,
 19    begin: Union[datetime, int, None] = None,
 20    end: Union[datetime, int, None] = None,
 21    params: Optional[Dict[str, Any]] = None,
 22    chunk_interval: Union[timedelta, int, None] = None,
 23    debug: bool = False,
 24    **kwargs: Any
 25) -> SuccessTuple:
 26    """
 27    Copy a pipe to another instance.
 28
 29    Parameters
 30    ----------
 31    instance_keys: str
 32        The instance to which to copy this pipe.
 33
 34    sync: bool, default True
 35        If `True`, sync the source pipe's documents 
 36
 37    begin: Union[datetime, int, None], default None
 38        Beginning datetime value to pass to `Pipe.get_data()`.
 39
 40    end: Union[datetime, int, None], default None
 41        End datetime value to pass to `Pipe.get_data()`.
 42
 43    params: Optional[Dict[str, Any]], default None
 44        Parameters filter to pass to `Pipe.get_data()`.
 45
 46    chunk_interval: Union[timedelta, int, None], default None
 47        The size of chunks to retrieve from `Pipe.get_data()` for syncing.
 48
 49    kwargs: Any
 50        Additional flags to pass to `Pipe.get_data()` and `Pipe.sync()`, e.g. `workers`.
 51
 52    Returns
 53    -------
 54    A SuccessTuple indicating success.
 55    """
 56    if str(instance_keys) == self.instance_keys:
 57        return False, f"Cannot copy {self} to instance '{instance_keys}'."
 58
 59    begin, end = self.parse_date_bounds(begin, end)
 60
 61    new_pipe = mrsm.Pipe(
 62        self.connector_keys,
 63        self.metric_key,
 64        self.location_key,
 65        parameters=self.parameters.copy(),
 66        instance=instance_keys,
 67    )
 68
 69    new_pipe_is_registered = new_pipe.id is not None
 70
 71    metadata_method = new_pipe.edit if new_pipe_is_registered else new_pipe.register
 72    metadata_success, metadata_msg = metadata_method(debug=debug)
 73    if not metadata_success:
 74        return metadata_success, metadata_msg
 75
 76    if not self.exists(debug=debug):
 77        return True, f"{self} does not exist; nothing to sync."
 78
 79    original_as_iterator = kwargs.get('as_iterator', None)
 80    kwargs['as_iterator'] = True
 81
 82    chunk_generator = self.get_data(
 83        begin=begin,
 84        end=end,
 85        params=params,
 86        chunk_interval=chunk_interval,
 87        debug=debug,
 88        **kwargs
 89    )
 90
 91    if original_as_iterator is None:
 92        _ = kwargs.pop('as_iterator', None)
 93    else:
 94        kwargs['as_iterator'] = original_as_iterator
 95
 96    sync_success, sync_msg = new_pipe.sync(
 97        chunk_generator,
 98        begin=begin,
 99        end=end,
100        params=params,
101        debug=debug,
102        **kwargs
103    )
104    msg = (
105        f"Successfully synced {new_pipe}:\n{sync_msg}"
106        if sync_success
107        else f"Failed to sync {new_pipe}:\n{sync_msg}"
108    )
109    return sync_success, msg

Copy a pipe to another instance.

Parameters
  • instance_keys (str): The instance to which to copy this pipe.
  • sync (bool, default True): If True, sync the source pipe's documents
  • begin (Union[datetime, int, None], default None): Beginning datetime value to pass to Pipe.get_data().
  • end (Union[datetime, int, None], default None): End datetime value to pass to Pipe.get_data().
  • params (Optional[Dict[str, Any]], default None): Parameters filter to pass to Pipe.get_data().
  • chunk_interval (Union[timedelta, int, None], default None): The size of chunks to retrieve from Pipe.get_data() for syncing.
  • kwargs (Any): Additional flags to pass to Pipe.get_data() and Pipe.sync(), e.g. workers.
Returns
  • A SuccessTuple indicating success.
def get_sync_status(self, debug: bool = False) -> Dict[str, Any] | None: 
39def get_sync_status(self, debug: bool = False) -> Optional[Dict[str, Any]]:
40    """
41    Return how this pipe's latest recorded sync went.
42
43    Every sync records its outcome on the pipe's instance: `mrsm sync pipes`, background jobs,
44    Pipeline runs, the API server and `pipe.sync()`. The outcome is written in the background
45    (a sync never waits for it): successes of one pipe at most every few seconds, failures at
46    once. A read in the same process sees the latest sync, and a script writes its last
47    outcomes when it exits.
48
49    Parameters
50    ----------
51    debug: bool, default False
52        Verbosity toggle.
53
54    Returns
55    -------
56    A dictionary (see `meerschaum.core.Pipe._status`), or `None` if no sync of this pipe has
57    been recorded. Reading the status from an unreachable instance raises an exception.
58
59    Examples
60    --------
61    >>> pipe = mrsm.Pipe('sql:main', 'weather', instance='sql:local')
62    >>> pipe.sync([{'ts': '2026-01-01', 'temp': 20.1}])
63    >>> status = pipe.get_sync_status()
64    >>> status['success'], status['consecutive_failures']
65    (True, 0)
66    """
67    from meerschaum import _core
68    return _core.pipe_get_sync_status(self, _status_context(self, debug=debug), debug=debug)

Return how this pipe's latest recorded sync went.

Every sync records its outcome on the pipe's instance: mrsm sync pipes, background jobs, Pipeline runs, the API server and pipe.sync(). The outcome is written in the background (a sync never waits for it): successes of one pipe at most every few seconds, failures at once. A read in the same process sees the latest sync, and a script writes its last outcomes when it exits.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
  • A dictionary (see meerschaum.core.Pipe._status), or None if no sync of this pipe has
  • been recorded. Reading the status from an unreachable instance raises an exception.
Examples
>>> pipe = mrsm.Pipe('sql:main', 'weather', instance='sql:local')
>>> pipe.sync([{'ts': '2026-01-01', 'temp': 20.1}])
>>> status = pipe.get_sync_status()
>>> status['success'], status['consecutive_failures']
(True, 0)
access: Dict[str, Any] 
33@property
34def access(self) -> Dict[str, Any]:
35    """
36    The pipe's access: `{'owner', 'team', 'visibility', 'edit'}` (`team` `None` is
37    `Everyone`). Change it with `set_access()`.
38    """
39    from meerschaum.core.access import get_pipe_access
40    return get_pipe_access(_raw_parameters(self))

The pipe's access: {'owner', 'team', 'visibility', 'edit'} (team None is Everyone). Change it with set_access().

owner: str | None 
43@property
44def owner(self) -> Optional[str]:
45    """The username of the pipe's owner (`None`: no owner)."""
46    return self.access.get('owner', None)

The username of the pipe's owner (None: no owner).

team: str 
49@property
50def team(self) -> str:
51    """The name of the pipe's team (`'Everyone'` when it names none)."""
52    from meerschaum.core.access import EVERYONE
53    return self.access.get('team', None) or EVERYONE

The name of the pipe's team ('Everyone' when it names none).

visibility: str 
56@property
57def visibility(self) -> str:
58    """Who can see the pipe: `'team'` (default), `'private'` or `'everyone'`."""
59    return self.access.get('visibility', 'team')

Who can see the pipe: 'team' (default), 'private' or 'everyone'.

edit_mode: str 
62@property
63def edit_mode(self) -> str:
64    """Who can change the pipe: `'team'` (default: anyone who can see it) or `'owner'`."""
65    return self.access.get('edit', 'team')

Who can change the pipe: 'team' (default: anyone who can see it) or 'owner'.

locks: List[str] 
68@property
69def locks(self) -> List[str]:
70    """
71    The pipe's locks (`parameters['locks']`): `'parameters'`, `'sync'`, `'clear'`, `'drop'`,
72    `'delete'` or `'all'`. Change them with `lock()` and `unlock()`.
73    """
74    from meerschaum.core.access import get_locks
75    return get_locks(_raw_parameters(self))

The pipe's locks (parameters['locks']): 'parameters', 'sync', 'clear', 'drop', 'delete' or 'all'. Change them with lock() and unlock().

def lock(self, *locks: str, debug: bool = False) -> Tuple[bool, str]: 
107def lock(self, *locks: str, debug: bool = False) -> mrsm.SuccessTuple:
108    """
109    Lock the pipe against writes. Every write path refuses what a lock refuses, locally too:
110
111    - `'parameters'`: parameter edits (changing `locks` stays allowed);
112    - `'sync'`: syncs, replace windows, deduplication and verification syncs;
113    - `'clear'`: row deletes (clears, replace windows, rebuilds);
114    - `'drop'`: drops, rebuilds and table schema changes (`verify schema`, repartitioning);
115    - `'delete'`: deleting (unregistering) the pipe;
116    - `'all'` (the default): all of them.
117
118    A registered pipe is saved at once; an unregistered one keeps the locks in memory until it
119    is registered.
120
121    Parameters
122    ----------
123    *locks: str
124        The locks to add (names or comma-separated text). Without any, `'all'`.
125
126    Returns
127    -------
128    A `SuccessTuple`.
129
130    Examples
131    --------
132    >>> pipe.lock('drop', 'delete')
133    (True, "Locked Pipe('sql:main', 'weather') (drop, delete).")
134    >>> pipe.clear()
135    (True, 'Success')
136    >>> pipe.drop()
137    (False, "Cannot drop Pipe('sql:main', 'weather'): it is locked (drop). Unlock it with ...")
138    """
139    from meerschaum.core.access import parse_locks, locks_with
140    try:
141        wanted = parse_locks(list(locks))
142    except ValueError as e:
143        return False, str(e)
144    new = locks_with(self.locks, wanted)
145    if not _is_registered(self, debug=debug):
146        _set_in_memory(self, 'locks', new)
147        return True, f"Locked {self} ({', '.join(new)}); saved when it is registered."
148    result = self.instance_connector.lock_pipe(self, wanted, debug=debug)
149    if result[0]:
150        ### The in-memory parameters follow (they patch the registered ones on reads).
151        _set_in_memory(self, 'locks', new)
152    _after_write(self, debug=debug)
153    return tuple(result)

Lock the pipe against writes. Every write path refuses what a lock refuses, locally too:

  • 'parameters': parameter edits (changing locks stays allowed);
  • 'sync': syncs, replace windows, deduplication and verification syncs;
  • 'clear': row deletes (clears, replace windows, rebuilds);
  • 'drop': drops, rebuilds and table schema changes (verify schema, repartitioning);
  • 'delete': deleting (unregistering) the pipe;
  • 'all' (the default): all of them.

A registered pipe is saved at once; an unregistered one keeps the locks in memory until it is registered.

Parameters
  • *locks (str): The locks to add (names or comma-separated text). Without any, 'all'.
Returns
Examples
>>> pipe.lock('drop', 'delete')
(True, "Locked Pipe('sql:main', 'weather') (drop, delete).")
>>> pipe.clear()
(True, 'Success')
>>> pipe.drop()
(False, "Cannot drop Pipe('sql:main', 'weather'): it is locked (drop). Unlock it with ...")
def unlock(self, *locks: str, debug: bool = False) -> Tuple[bool, str]: 
156def unlock(self, *locks: str, debug: bool = False) -> mrsm.SuccessTuple:
157    """
158    Remove locks from the pipe (without any, every lock). Removing one kind from `'all'` keeps
159    the other kinds.
160
161    Parameters
162    ----------
163    *locks: str
164        The locks to remove (names or comma-separated text). Without any, `'all'`.
165
166    Returns
167    -------
168    A `SuccessTuple`.
169    """
170    from meerschaum.core.access import parse_locks, locks_without
171    try:
172        removed = parse_locks(list(locks))
173    except ValueError as e:
174        return False, str(e)
175    new = locks_without(self.locks, removed)
176    if not _is_registered(self, debug=debug):
177        _set_in_memory(self, 'locks', new or None)
178        return True, f"Unlocked {self}" + (f" (still locked: {', '.join(new)})." if new else ".")
179    result = self.instance_connector.unlock_pipe(self, removed, debug=debug)
180    if result[0]:
181        _set_in_memory(self, 'locks', new or None)
182    _after_write(self, debug=debug)
183    return tuple(result)

Remove locks from the pipe (without any, every lock). Removing one kind from 'all' keeps the other kinds.

Parameters
  • *locks (str): The locks to remove (names or comma-separated text). Without any, 'all'.
Returns
def set_access( self, team: Any = <object object>, visibility: Any = <object object>, edit: Any = <object object>, owner: Any = <object object>, debug: bool = False) -> Tuple[bool, str]: 
186def set_access(
187    self,
188    team: Any = _UNSET,
189    visibility: Any = _UNSET,
190    edit: Any = _UNSET,
191    owner: Any = _UNSET,
192    debug: bool = False,
193) -> mrsm.SuccessTuple:
194    """
195    Set who owns the pipe and who may see and change it (`parameters['access']`). Arguments not
196    given keep their current values; `None` clears the owner or the team (back to
197    `Everyone`).
198
199    The API server, the MCP server and the web console enforce access; through an `api:`
200    instance only the pipe's owner, its team's managers and administrators may change it. A
201    `parameters` lock refuses the change.
202
203    Parameters
204    ----------
205    team: Optional[str]
206        The pipe's team (`None` or `'Everyone'`: every user's).
207
208    visibility: str
209        `'team'` (its team's members and its owner), `'private'` (its owner, its team's
210        managers and administrators) or `'everyone'` (every user).
211
212    edit: str
213        `'team'` (anyone who can see it may change it within their permissions) or `'owner'`
214        (its owner, its team's managers and administrators).
215
216    owner: Optional[str]
217        The owner's username (`None`: no owner).
218
219    Returns
220    -------
221    A `SuccessTuple`.
222
223    Examples
224    --------
225    >>> pipe.set_access(team='ops', visibility='private')
226    (True, "Set the access of Pipe('sql:main', 'weather'): team ops, visibility private, edit team.")
227    >>> pipe.team, pipe.visibility
228    ('ops', 'private')
229    """
230    from meerschaum.core.access import parse_access
231    current = dict(self.access)
232    new = dict(current)
233    if team is not _UNSET:
234        new['team'] = team
235    if visibility is not _UNSET:
236        new['visibility'] = visibility
237    if edit is not _UNSET:
238        new['edit'] = edit
239    if owner is not _UNSET:
240        new['owner'] = owner
241    try:
242        new = parse_access(new)
243    except ValueError as e:
244        return False, str(e)
245    if not _is_registered(self, debug=debug):
246        _set_in_memory(self, 'access', new)
247        return True, f"Set the access of {self}; saved when it is registered."
248    result = self.instance_connector.set_pipe_access(self, new, debug=debug)
249    if result[0]:
250        _set_in_memory(self, 'access', new)
251    _after_write(self, debug=debug)
252    return tuple(result)

Set who owns the pipe and who may see and change it (parameters['access']). Arguments not given keep their current values; None clears the owner or the team (back to Everyone).

The API server, the MCP server and the web console enforce access; through an api: instance only the pipe's owner, its team's managers and administrators may change it. A parameters lock refuses the change.

Parameters
  • team (Optional[str]): The pipe's team (None or 'Everyone': every user's).
  • visibility (str): 'team' (its team's members and its owner), 'private' (its owner, its team's managers and administrators) or 'everyone' (every user).
  • edit (str): 'team' (anyone who can see it may change it within their permissions) or 'owner' (its owner, its team's managers and administrators).
  • owner (Optional[str]): The owner's username (None: no owner).
Returns
Examples
>>> pipe.set_access(team='ops', visibility='private')
(True, "Set the access of Pipe('sql:main', 'weather'): team ops, visibility private, edit team.")
>>> pipe.team, pipe.visibility
('ops', 'private')
class Pipeline: 
 588class Pipeline:
 589    """
 590    A selection over the pipe graph: member pipes plus traversal depths (design §5.2, D4).
 591
 592    Parameters
 593    ----------
 594    pipes: Union[str, mrsm.Pipe, Iterable[...], None], default None
 595        What to select:
 596
 597        - a `str` without `|` is the **name of a saved Pipeline**, loaded from `instance`
 598          (`mrsm.Pipeline('daily')`; `ValueError` if it does not exist);
 599        - a `str` with `|` is a pipe reference, `'instance|connector|metric|location'`;
 600        - a `Pipe`, or a `(connector_keys, metric_key[, location_key])` tuple, is one member;
 601        - a list may mix all of these: the members of every named Pipeline plus the given
 602          pipes (the dictionary `mrsm.get_pipes()` returns works too).
 603
 604    parents: Union[bool, int, None], default None
 605        `True` includes every ancestor of the members; an `int` includes that many generations.
 606        For a saved Pipeline, `None` keeps its saved setting and `False` turns traversal off.
 607
 608    children: Union[bool, int, None], default None
 609        `True` includes every descendant of the members; an `int` that many generations.
 610        For a saved Pipeline, `None` keeps its saved setting and `False` turns traversal off.
 611
 612    tags, connector_keys, metric_keys, location_keys: Union[str, List[str], None], default None
 613        Select the members by the same filters as `-t`, `-c`, `-m` and `-l`
 614        (negate a value with a leading `_`). Used when `pipes` is not provided.
 615
 616    instance: Union[str, InstanceConnector, None], default None
 617        The instance that keeps the engine state and the saved Pipelines (default: the member
 618        pipes' instance, else the configured one). Saved Pipelines are loaded from here.
 619
 620    name: Optional[str], default None
 621        The name to save this selection under (`save()`), also used in its run IDs.
 622        Without `pipes` or filters, `Pipeline(name='daily')` loads the saved Pipeline.
 623
 624    graph: Optional[bool], default None
 625        Whether the run respects the dependencies between the members even without traversal
 626        (a child member waits for its parent member and is capped at its watermark).
 627        A saved Pipeline keeps its `graph` setting (default `True`), like
 628        `mrsm sync pipelines NAME`; an unsaved selection without `parents` / `children` syncs
 629        each member on its own, like `mrsm sync pipes`.
 630
 631    Examples
 632    --------
 633    >>> pipeline = mrsm.Pipeline(pipe, parents=True)
 634    >>> plan = pipeline.plan()
 635    >>> run = pipeline.sync(workers=4)
 636    >>> run.success, run.run_id, run.inserted
 637    >>> pipeline.save('daily')
 638    >>> mrsm.Pipeline('daily').sync()
 639    """
 640
 641    def __init__(
 642        self,
 643        pipes: Union[str, mrsm.Pipe, Iterable[Any], None] = None,
 644        parents: Union[bool, int, None] = None,
 645        children: Union[bool, int, None] = None,
 646        tags: Union[str, List[str], None] = None,
 647        connector_keys: Union[str, List[str], None] = None,
 648        metric_keys: Union[str, List[str], None] = None,
 649        location_keys: Union[str, List[str], None] = None,
 650        instance: Union[str, Any, None] = None,
 651        name: Optional[str] = None,
 652        graph: Optional[bool] = None,
 653    ):
 654        if name is not None and not isinstance(name, str):
 655            raise TypeError(f"A Pipeline name must be a string (got {name!r}).")
 656        names, targets = _split_selection(pipes)
 657        filters = {
 658            'tags': tags,
 659            'connector_keys': connector_keys,
 660            'metric_keys': metric_keys,
 661            'location_keys': location_keys,
 662        }
 663        filters = {k: ([v] if isinstance(v, str) else v) for k, v in filters.items() if v}
 664        if not names and not targets and not filters:
 665            if not name:
 666                raise ValueError(
 667                    "A Pipeline needs member pipes, a tags / keys filter or the name of a saved "
 668                    "Pipeline, e.g. `mrsm.Pipeline(pipe)`, `mrsm.Pipeline(tags=['daily'])` or "
 669                    "`mrsm.Pipeline('daily')`."
 670                )
 671            names = [name]
 672        if names and filters:
 673            raise ValueError(
 674                "A saved Pipeline cannot be combined with a tags / keys filter: "
 675                "pass the extra pipes themselves."
 676            )
 677        definitions = []
 678        registry = None
 679        if names:
 680            registry = _registry_of(instance)
 681            seen = set()
 682            for saved_name in names:
 683                if saved_name in seen:
 684                    continue
 685                seen.add(saved_name)
 686                definitions.append(_read_definition(saved_name, registry))
 687        self._configure(
 688            definitions,
 689            targets,
 690            filters,
 691            parents=parents,
 692            children=children,
 693            instance=(registry if registry is not None else instance),
 694            name=name,
 695            graph=graph,
 696        )
 697
 698    def _configure(
 699        self,
 700        definitions: List[Dict[str, Any]],
 701        targets: List[Any],
 702        filters: Dict[str, Any],
 703        parents: Union[bool, int, None] = None,
 704        children: Union[bool, int, None] = None,
 705        instance: Union[str, Any, None] = None,
 706        name: Optional[str] = None,
 707        graph: Optional[bool] = None,
 708    ) -> None:
 709        """Set the selection: saved definitions (in order), extra members and filters."""
 710        names = [definition['name'] for definition in definitions]
 711        ### A member given as a `Pipe` object keeps that object (and its in-memory parameters);
 712        ### the other members of a saved Pipeline are built from their references.
 713        given = {
 714            _pipe_ref(target): target
 715            for target in targets
 716            if isinstance(target, mrsm.Pipe)
 717        }
 718        members: List[Any] = []
 719        seen = set()
 720        loaded: Dict[str, mrsm.Pipe] = {}
 721        for definition in definitions:
 722            for ref in definition.get('targets', None) or []:
 723                if ref in seen:
 724                    continue
 725                seen.add(ref)
 726                if ref in given:
 727                    members.append(given[ref])
 728                    continue
 729                member = _pipe_from_ref(ref)
 730                members.append(member)
 731                loaded[ref] = member
 732        for target in targets:
 733            if isinstance(target, mrsm.Pipe):
 734                ref = _pipe_ref(target)
 735                if ref in seen:
 736                    continue
 737                seen.add(ref)
 738            members.append(target)
 739        self._definitions: List[Dict[str, Any]] = [deepcopy(d) for d in definitions]
 740        self._loaded = loaded
 741        self._sources: List[Any] = list(names) + list(targets)
 742        self._targets: List[Any] = members
 743        self._filters: Dict[str, Any] = filters
 744        self.parents = (
 745            get_depth(parents)
 746            if parents is not None or not definitions
 747            else _common([get_depth(d.get('parents')) for d in definitions], 'parents', names)
 748        )
 749        self.children = (
 750            get_depth(children)
 751            if children is not None or not definitions
 752            else _common([get_depth(d.get('children')) for d in definitions], 'children', names)
 753        )
 754        self._graph: Optional[bool] = None if graph is None else bool(graph)
 755        if name is None and len(definitions) == 1 and not targets:
 756            name = definitions[0]['name']
 757        self.name: Optional[str] = name
 758        if definitions:
 759            self.instance = _registry_of(instance)
 760        else:
 761            pipe_targets = [target for target in members if isinstance(target, mrsm.Pipe)]
 762            self.instance = _instance_of(pipe_targets, instance)
 763
 764    @classmethod
 765    def _from_definitions(
 766        cls,
 767        definitions: List[Dict[str, Any]],
 768        instance: Any,
 769        name: Optional[str] = None,
 770    ) -> 'Pipeline':
 771        """A Pipeline bound to saved definitions which were already read."""
 772        pipeline = cls.__new__(cls)
 773        pipeline._configure(definitions, [], {}, instance=instance, name=name)
 774        return pipeline
 775
 776    @classmethod
 777    def load(cls, name: str, instance=None) -> 'Pipeline':
 778        """
 779        Load the saved Pipeline `name` from `instance` (default: the configured instance).
 780        The same as `mrsm.Pipeline(name, instance=instance)`.
 781        """
 782        return cls(name, instance=instance)
 783
 784    @property
 785    def definition(self) -> Optional[Dict[str, Any]]:
 786        """
 787        The saved definition this Pipeline was loaded from or saved as (`name`, `registry`,
 788        `id`, `revision`, `targets`, `parents`, `children`, `description`, `defaults`, ...),
 789        or `None` for an unsaved selection (or one combined from several saved Pipelines).
 790        """
 791        return self._definitions[0] if len(self._definitions) == 1 else None
 792
 793    @definition.setter
 794    def definition(self, value: Optional[Dict[str, Any]]) -> None:
 795        self._bind([value] if value else [])
 796
 797    def _bind(self, definitions: List[Dict[str, Any]]) -> None:
 798        """Bind this object to saved definitions (after a save, a rename, a refresh, ...)."""
 799        self._definitions = list(definitions)
 800
 801    def _owner_of(self, pipe: Any) -> Optional[str]:
 802        """
 803        The name of the saved Pipeline that `pipe` was loaded from as a member reference
 804        (`None` for a pipe given as a `Pipe` object, or one which is not a member).
 805        A member loaded from a reference must be registered already: registering it would
 806        create an empty pipe, so `sync()` refuses instead.
 807        """
 808        if not isinstance(pipe, mrsm.Pipe):
 809            return None
 810        ref = _pipe_ref(pipe)
 811        if self._loaded.get(ref, None) is not pipe:
 812            return None
 813        for definition in self._definitions:
 814            if ref in (definition.get('targets', None) or []):
 815                return definition.get('name', None)
 816        return self.name or ''
 817
 818    @property
 819    def graph(self) -> bool:
 820        """
 821        Whether runs respect the dependencies between members even without traversal
 822        (see the `graph` argument).
 823        """
 824        if self._graph is not None:
 825            return self._graph
 826        if not self._definitions:
 827            return False
 828        return _common([d.get('graph', True) for d in self._definitions],
 829                       'graph', [d['name'] for d in self._definitions])
 830
 831    @graph.setter
 832    def graph(self, value: Optional[bool]) -> None:
 833        self._graph = None if value is None else bool(value)
 834
 835    @property
 836    def members(self) -> List[Any]:
 837        """
 838        The member pipes (or keys) the selection starts from, before traversal
 839        (empty for a tags / keys filter). Also available as `targets`.
 840
 841        Assign a new list (`Pipe` objects, key tuples or `pipe_ref` strings) to change the
 842        selection, then `save()` to keep the change:
 843
 844        ```python
 845        daily = mrsm.Pipeline('daily')
 846        daily.members = daily.members + [mrsm.Pipe('sql:main', 'weekly')]
 847        daily.save()
 848        ```
 849        """
 850        return list(self._targets)
 851
 852    @members.setter
 853    def members(self, pipes: Any) -> None:
 854        names, targets = _split_selection(pipes)
 855        if names:
 856            raise ValueError(
 857                "Members are pipes (Pipe objects, key tuples or pipe references), not the names "
 858                "of saved Pipelines: combine saved Pipelines with `mrsm.Pipeline([...])`."
 859            )
 860        members: List[Any] = []
 861        seen = set()
 862        for target in targets:
 863            if isinstance(target, mrsm.Pipe):
 864                ref = _pipe_ref(target)
 865                if ref in seen:
 866                    continue
 867                seen.add(ref)
 868            members.append(target)
 869        self._loaded = {
 870            ref: member
 871            for ref, member in self._loaded.items()
 872            if any(target is member for target in members)
 873        }
 874        self._targets = members
 875        self._filters = {}
 876        self._sources = list(members)
 877
 878    @property
 879    def targets(self) -> List[Any]:
 880        """The member pipes (or keys) the selection starts from (the same as `members`)."""
 881        return list(self._targets)
 882
 883    @targets.setter
 884    def targets(self, pipes: Any) -> None:
 885        self.members = pipes
 886
 887    @property
 888    def pipes(self) -> List[mrsm.Pipe]:
 889        """
 890        Every pipe the selection resolves to (members plus the traversed parents and children),
 891        in dependency order. This plans the selection, registering nothing: member pipes that
 892        are not registered yet come last (their dependencies are unknown until they are).
 893        """
 894        plan = self.plan()
 895        planned = plan.pipes
 896        return planned + [pipe for pipe in plan.unregistered if pipe not in planned]
 897
 898    def _unregistered(self) -> List[mrsm.Pipe]:
 899        """Member pipes that are not registered on their instance (temporary pipes excluded)."""
 900        return [
 901            target
 902            for target in self._targets
 903            if isinstance(target, mrsm.Pipe)
 904            and not target.temporary
 905            and target.get_id() is None
 906        ]
 907
 908    def _execution_defaults(self, explicit: Dict[str, Any]) -> Dict[str, Any]:
 909        """The saved execution defaults of the bound definitions (explicit options win)."""
 910        from meerschaum.core.Pipeline._saved import validate_execution_defaults
 911        merged: Dict[str, Any] = {}
 912        owners: Dict[str, str] = {}
 913        for definition in self._definitions:
 914            defaults = validate_execution_defaults(definition.get('defaults', None) or {})
 915            for key, value in defaults.items():
 916                if key in merged and merged[key] != value and key not in explicit:
 917                    raise ValueError(
 918                        f"The saved Pipelines '{owners[key]}' and '{definition['name']}' have "
 919                        f"different defaults for `{key}`; pass `{key}=` to choose one."
 920                    )
 921                if key not in merged:
 922                    merged[key] = value
 923                    owners[key] = definition['name']
 924        return merged
 925
 926    def _request(self, targets: Optional[List[Any]] = None, **kwargs: Any) -> Dict[str, Any]:
 927        if self._definitions:
 928            kwargs = {**self._execution_defaults(kwargs), **kwargs}
 929            kwargs["pipeline_definitions"] = deepcopy(self._definitions)
 930        run_name = self.name or ('combined' if len(self._definitions) > 1 else None)
 931        if run_name:
 932            kwargs.setdefault("run_name", run_name)
 933        if self.graph:
 934            kwargs.setdefault("graph", True)
 935        parents = kwargs.pop('parents', self.parents)
 936        children = kwargs.pop('children', self.children)
 937        for reserved in ('instance', 'targets', 'filters'):
 938            if reserved in kwargs:
 939                raise ValueError(f"Set Pipeline {reserved} on the definition, not as an execution option.")
 940        return build_request(
 941            self.instance,
 942            targets=(self._targets if targets is None else targets),
 943            filters=self._filters,
 944            parents=parents,
 945            children=children,
 946            **kwargs
 947        )
 948
 949    def _register_targets(self) -> Optional[SuccessTuple]:
 950        """The engine reads the graph from the instance, so register the members first."""
 951        for target in self._targets:
 952            if not isinstance(target, mrsm.Pipe) or target.temporary:
 953                continue
 954            if target.get_id() is None:
 955                owner = self._owner_of(target)
 956                if owner is not None:
 957                    return False, (
 958                        f"{target} is a member of Pipeline '{owner}' but is not registered "
 959                        f"on '{target.instance_keys}'. Register it, or remove it from the "
 960                        "Pipeline."
 961                    )
 962                success, message = target.register()
 963                if not success:
 964                    return False, message
 965        return None
 966
 967    def plan(
 968        self,
 969        begin: Any = _UNSET_BOUND,
 970        end: Any = _UNSET_BOUND,
 971        expected: bool = False,
 972        **kwargs: Any
 973    ) -> Plan:
 974        """
 975        Plan the run without moving any data (and without registering anything).
 976
 977        Parameters
 978        ----------
 979        begin, end: Union[datetime, int, str, None], default None
 980            Optionally bound the sync window. `None` and `''` leave it to the planner
 981            (a root syncs from its sync time minus the backtrack).
 982
 983        expected: bool, default False
 984            If `False`, every node is planned against the state as it is now: a node whose
 985            parents run first is provisional (`*`) and is planned again by the executor once
 986            they finish. If `True`, return what `mrsm plan pipes` prints: each provisional node
 987            re-planned as if every node that runs before it synced new rows after its
 988            watermark, so its window starts where the executor will start it (often the start
 989            of a parent's chunk-grid cell). Only the start is predicted; the executor caps the
 990            end at the parents' new watermarks.
 991
 992        Returns
 993        -------
 994        A `Plan` (`str(plan)` is the table; `plan.nodes` is the data). Member pipes that are
 995        not registered yet are in `plan.unregistered` and named in `plan.warnings`.
 996        """
 997        options = {key: value for key, value in {'begin': begin, 'end': end, **kwargs}.items()
 998                   if value is not _UNSET_BOUND}
 999        unregistered = self._unregistered()
1000        request = self._request(
1001            targets=(
1002                [target for target in self._targets if target not in unregistered]
1003                if unregistered
1004                else None
1005            ),
1006            **options
1007        )
1008        if expected:
1009            request['expected'] = True
1010        data = call('engine_plan', request)
1011        return Plan(
1012            data,
1013            pipeline=self,
1014            options={
1015                key: value
1016                for key, value in options.items()
1017                if value not in (None, '')
1018            },
1019            unregistered=unregistered,
1020        )
1021
1022    def sync(self, begin: Any = _UNSET_BOUND, end: Any = _UNSET_BOUND, **kwargs: Any) -> RunResult:
1023        """
1024        Plan and run the selection, registering member pipes that are not registered yet
1025        (a saved Pipeline's members must already be registered).
1026
1027        Parameters
1028        ----------
1029        begin, end: Union[datetime, int, str, None]
1030            Bound the sync window of the root nodes (other nodes are clamped to it).
1031            `None` and `''` leave the window to the planner: a root syncs from its sync time
1032            minus the backtrack, as a bare `pipe.sync()` does by default. (A bare
1033            `pipe.sync(begin=None)` means "no lower bound"; in an engine run it does not.)
1034
1035        workers: Optional[int], default None
1036            Run up to this many nodes at once (default `system.engine.workers`).
1037
1038        force, retries, min_seconds, check_existing, enforce_dtypes, chunksize, params, yes:
1039            The `Pipe.sync()` options, applied to every node. `yes` (or `force`) confirms a
1040            `rebuild`; without it a node whose definition changed is left `rebuild_pending`.
1041
1042        resume: Optional[str], default None
1043            Re-plan only the nodes that failed or never ran in this earlier run ID.
1044
1045        on_event: Optional[Callable[[Dict[str, Any]], Any]], default None
1046            Called with every event of the run as it happens (`run_started`, `node_planned`,
1047            `node_started`, `chunk_started`, `chunk_finished`, `node_finished`, `node_failed`,
1048            `node_skipped`, `watermark_advanced`, ..., `run_finished`), each a `dict` with
1049            `event`, `run_id`, `seq`, `ts` (an aware UTC `datetime`), `pipe_ref` and the
1050            event's own fields. It runs on this thread; the sync work runs on the engine's
1051            threads without the GIL. If it raises an `Exception`, the run continues, the error
1052            is warned about once and kept in `RunResult.callback_error`, and no further events
1053            are delivered. `KeyboardInterrupt` cancels the run.
1054
1055        Returns
1056        -------
1057        A `RunResult`: a `SuccessTuple` with `run_id`, `plan`, `outcomes`, `events()`, the
1058        totals `inserted`, `updated`, `upserted` and `deleted` over every node, and `duration`
1059        (the run's wall-clock seconds).
1060        """
1061        on_event = kwargs.pop('on_event', None)
1062        failure = self._register_targets()
1063        if failure is not None:
1064            return RunResult(*failure)
1065        options = {key: value for key, value in {'begin': begin, 'end': end, **kwargs}.items()
1066                   if value is not _UNSET_BOUND}
1067        try:
1068            request = self._request(**options)
1069        except ValueError as e:
1070            return RunResult(False, str(e))
1071        callback = EventCallback(on_event) if on_event is not None else None
1072        if callback is None:
1073            data = call('engine_run', request)
1074        else:
1075            data = call('engine_run', request, on_event=callback)
1076            if callback.warning is not None:
1077                data['warnings'] = list(data.get('warnings', None) or []) + [callback.warning]
1078        run = RunResult(data['success'], data['message'], data, pipeline=self)
1079        run._instance_request = self._instance_request()
1080        if callback is not None:
1081            run.callback_error = callback.error
1082        return run
1083
1084    def run(self, begin: Any = _UNSET_BOUND, end: Any = _UNSET_BOUND, **kwargs: Any) -> RunResult:
1085        """An alias of `Pipeline.sync()`."""
1086        return self.sync(begin=begin, end=end, **kwargs)
1087
1088    def _instance_request(self) -> Dict[str, Any]:
1089        return build_request(self.instance)
1090
1091    def _members_to_save(self) -> List[Any]:
1092        """The members a saved definition stores: the members, or the pipes the filters match."""
1093        if self._targets:
1094            return list(self._targets)
1095        if not self._filters:
1096            return []
1097        return [
1098            pipe
1099            for pipe in mrsm.get_pipes(
1100                connector_keys=self._filters.get('connector_keys', None),
1101                metric_keys=self._filters.get('metric_keys', None),
1102                location_keys=self._filters.get('location_keys', None),
1103                tags=self._filters.get('tags', None),
1104                instance=self.instance,
1105                as_list=True,
1106            )
1107            if str(pipe.connector_keys) != 'mrsm'
1108        ]
1109
1110    def _settings_to_carry(self, with_defaults: bool = True) -> Dict[str, Any]:
1111        """The description, defaults and layout a new saved definition starts from."""
1112        definition = self.definition
1113        if definition is not None:
1114            return {
1115                'description': definition.get('description', None),
1116                'defaults': definition.get('defaults', None) if with_defaults else None,
1117                'layout': definition.get('layout', None),
1118            }
1119        if self._definitions and with_defaults:
1120            try:
1121                return {'defaults': self._execution_defaults({})}
1122            except ValueError as e:
1123                raise ValueError(f"{e} (pass `defaults={{...}}` to `save()`)") from None
1124        return {}
1125
1126    def save(
1127        self,
1128        name: Optional[str] = None,
1129        overwrite: bool = False,
1130        description: Optional[str] = None,
1131        defaults: Optional[Dict[str, Any]] = None,
1132    ) -> SuccessTuple:
1133        """
1134        Save this selection's members, traversal and execution mode under `name`
1135        (default: `self.name`). Saving never changes the members' tags or registrations.
1136
1137        Parameters
1138        ----------
1139        name: Optional[str], default None
1140            The name to save under (letters, numbers, dots, underscores and hyphens).
1141
1142        overwrite: bool, default False
1143            A different Pipeline already saved under `name` is only replaced with
1144            `overwrite=True` (its description, defaults and layout are kept unless passed).
1145            A Pipeline loaded from `name` saves its changes without it, unless someone else
1146            saved a newer revision since it was loaded (load it again). Saving the same
1147            members, traversal and execution mode again is a no-op success.
1148
1149        description: Optional[str], default None
1150            Set the saved description.
1151
1152        defaults: Optional[Dict[str, Any]], default None
1153            Set the saved execution defaults (`begin`, `end`, `workers`, `retries`,
1154            `min_seconds`, `chunksize`, `check_existing`, `enforce_dtypes`, `force`, `params`,
1155            `yes`), used by every run of the saved Pipeline unless a run passes its own.
1156
1157        Returns
1158        -------
1159        A `SuccessTuple`. On success, `self.name` and `self.definition` are the saved ones.
1160        """
1161        from meerschaum.core.Pipeline._saved import (
1162            _read, _refs, registration, save_pipeline, validate_name,
1163        )
1164        name = name or self.name
1165        if not name:
1166            return False, "Name the Pipeline to save it, e.g. `pipeline.save('daily')`."
1167        try:
1168            validate_name(name)
1169        except ValueError as e:
1170            return False, f"Cannot save Pipeline {name!r}: {e}"
1171
1172        registry = self.instance
1173        try:
1174            members = self._members_to_save()
1175            graph = True if self._graph is None and not self._definitions else self.graph
1176            registration_pipe = registration(name, registry)
1177            registry_keys = registration_pipe.instance_keys
1178            exists = registration_pipe.get_id() is not None
1179            source = self.definition
1180            editing = (
1181                exists
1182                and source is not None
1183                and source.get('name', None) == name
1184                and str(source.get('registry', None)) == str(registry_keys)
1185            )
1186            expected_revision = None
1187            if exists and not overwrite:
1188                ### Saving what is already saved is a no-op (no new revision for other editors
1189                ### to trip over). Otherwise a Pipeline loaded from `name` saves its changes
1190                ### against the revision it loaded, and replacing a different one needs
1191                ### `overwrite`.
1192                current = _read(name, registry)
1193                same_selection = (
1194                    sorted(current.get('targets', None) or []) == _refs(members, registry_keys)
1195                    and get_depth(current.get('parents', None)) == self.parents
1196                    and get_depth(current.get('children', None)) == self.children
1197                    and current.get('graph', True) == graph
1198                )
1199                unchanged = (
1200                    (description is None or description == current.get('description', ''))
1201                    and (defaults is None or defaults == current.get('defaults', {}))
1202                )
1203                if same_selection and unchanged:
1204                    self._bind([current])
1205                    self.name = name
1206                    return True, (
1207                        f"Pipeline '{name}' is already saved on '{registry_keys}' "
1208                        "with these members."
1209                    )
1210                if not editing and not same_selection:
1211                    return False, (
1212                        f"A different Pipeline '{name}' is already saved on '{registry_keys}'. "
1213                        f"Load it with `mrsm.Pipeline({name!r})` to change it, or pass "
1214                        "`overwrite=True` to replace it."
1215                    )
1216                expected_revision = source['revision'] if editing else current['revision']
1217
1218            if exists:
1219                definition = save_pipeline(
1220                    name, members, registry, self.parents, self.children,
1221                    expected_revision=expected_revision,
1222                    description=description,
1223                    defaults=defaults,
1224                    graph=graph,
1225                )
1226            else:
1227                carried = self._settings_to_carry(with_defaults=(defaults is None))
1228                definition = save_pipeline(
1229                    name, members, registry, self.parents, self.children,
1230                    description=(description if description is not None else carried.get('description', None)),
1231                    defaults=(defaults if defaults is not None else carried.get('defaults', None)),
1232                    layout=carried.get('layout', None),
1233                    create=True,
1234                    graph=graph,
1235                )
1236        except Exception as e:
1237            return False, f"Failed to save Pipeline '{name}' on '{registry}': {e}"
1238
1239        self._bind([definition])
1240        self.name = name
1241        num_members = len(definition.get('targets', None) or [])
1242        try:
1243            num_unregistered = len(self._unregistered())
1244        except Exception:
1245            num_unregistered = 0
1246        note = (
1247            f" {num_unregistered} of them {'is' if num_unregistered == 1 else 'are'} not "
1248            "registered yet: `sync()` registers them."
1249            if num_unregistered
1250            else ''
1251        )
1252        return True, (
1253            f"Saved Pipeline '{name}' on '{definition['registry']}' "
1254            f"({num_members} member{'s' if num_members != 1 else ''}, "
1255            f"revision {definition['revision']}).{note}"
1256        )
1257
1258    def exists(self) -> bool:
1259        """Whether a Pipeline is saved under this Pipeline's `name` on its instance."""
1260        if not self.name:
1261            return False
1262        from meerschaum.core.Pipeline._saved import registration
1263        try:
1264            return registration(self.name, self.instance).get_id() is not None
1265        except ValueError:
1266            return False
1267
1268    def delete(self) -> SuccessTuple:
1269        """
1270        Delete the saved Pipeline (its definition and the membership tags it owns).
1271        The member pipes and their data are kept, and this object stays usable as an unsaved
1272        selection. Scheduled jobs that sync the Pipeline must be removed first.
1273        """
1274        from meerschaum.core.Pipeline._saved import delete_pipeline
1275        if not self.name:
1276            return False, "This Pipeline has no name, so it is not saved."
1277        try:
1278            success, message = delete_pipeline(self.name, self.instance)
1279        except Exception as e:
1280            return False, f"Failed to delete Pipeline '{self.name}': {e}"
1281        if not success:
1282            return success, message
1283        self._bind([])
1284        return True, (
1285            f"Deleted Pipeline '{self.name}' from '{self.instance}' "
1286            "(its member pipes and their data are kept)."
1287        )
1288
1289    def rename(self, new_name: str) -> SuccessTuple:
1290        """
1291        Rename the saved Pipeline (its identity and run history are kept).
1292        Scheduled jobs that sync the Pipeline must be removed first.
1293        """
1294        from meerschaum.core.Pipeline._saved import rename_pipeline
1295        if not self.exists():
1296            return False, f"Pipeline '{self.name}' is not saved on '{self.instance}'."
1297        definition = self.definition
1298        expected_revision = (
1299            definition['revision']
1300            if definition is not None and definition.get('name', None) == self.name
1301            else None
1302        )
1303        old_name = self.name
1304        try:
1305            renamed = rename_pipeline(
1306                old_name, new_name, self.instance, expected_revision=expected_revision,
1307            )
1308        except Exception as e:
1309            return False, f"Failed to rename Pipeline '{old_name}': {e}"
1310        self._bind([renamed])
1311        self.name = new_name
1312        if old_name == new_name:
1313            return True, f"Pipeline '{old_name}' already has that name."
1314        return True, f"Renamed Pipeline '{old_name}' to '{new_name}' on '{self.instance}'."
1315
1316    def copy(self, new_name: str, instance: Union[str, Any, None] = None) -> SuccessTuple:
1317        """
1318        Save a copy of this Pipeline as `new_name`, on `instance` (default: this Pipeline's
1319        instance). The copy has the same members (which keep their own instances), traversal,
1320        description and defaults, and a new identity; this object is unchanged.
1321        """
1322        from meerschaum.core.Pipeline._saved import save_pipeline, validate_name
1323        try:
1324            validate_name(new_name)
1325        except ValueError as e:
1326            return False, f"Cannot copy to {new_name!r}: {e}"
1327        registry = _registry_of(instance) if instance is not None else self.instance
1328        try:
1329            carried = self._settings_to_carry()
1330            definition = save_pipeline(
1331                new_name, self._members_to_save(), registry, self.parents, self.children,
1332                description=carried.get('description', None),
1333                defaults=carried.get('defaults', None),
1334                layout=carried.get('layout', None),
1335                create=True,
1336            )
1337        except Exception as e:
1338            return False, f"Failed to copy to Pipeline '{new_name}' on '{registry}': {e}"
1339        source = f"Pipeline '{self.name}'" if self.name else 'this selection'
1340        return True, f"Copied {source} to Pipeline '{new_name}' on '{definition['registry']}'."
1341
1342    def analyze(self) -> Dict[str, Any]:
1343        """
1344        Resolve the saved Pipeline's members and traversal and preview the membership tags
1345        (`pipeline-NAME`) it would add or remove, without changing anything (like
1346        `mrsm analyze pipelines NAME`). Apply the proposal with `apply_analysis()`.
1347
1348        Returns
1349        -------
1350        The proposal: `resolved` (each pipe and why it is included), `changes` (tag edits),
1351        `shared` (pipes other Pipelines also use), `errors` and a `fingerprint`.
1352        """
1353        from meerschaum.core.Pipeline._saved import analyze_pipeline
1354        if not self.name:
1355            raise ValueError("Save the Pipeline before analyzing it, e.g. `pipeline.save('daily')`.")
1356        return analyze_pipeline(self.name, self.instance)
1357
1358    def apply_analysis(self, proposal: Optional[Dict[str, Any]] = None) -> SuccessTuple:
1359        """
1360        Apply the membership tag changes of a proposal from `analyze()` (without one, analyze
1361        now and apply that, like `mrsm analyze pipelines NAME --yes`). A proposal whose inputs
1362        changed since it was made is refused: analyze again and review the new one.
1363        """
1364        from meerschaum.core.Pipeline._saved import _read, apply_pipeline_analysis
1365        if not self.name:
1366            return False, "This Pipeline has no name, so it is not saved."
1367        try:
1368            proposal = proposal if proposal is not None else self.analyze()
1369            result = apply_pipeline_analysis(proposal, self.instance)
1370        except Exception as e:
1371            return False, f"Failed to apply the analysis of Pipeline '{self.name}': {e}"
1372        message = result.get('message', '')
1373        failures = [row for row in result.get('results', None) or [] if not row.get('success', True)]
1374        if failures:
1375            message += ''.join(f"\n{row['pipe_ref']}: {row['message']}" for row in failures)
1376        try:
1377            self._bind([_read(self.name, self.instance)])
1378        except Exception:
1379            pass
1380        return bool(result.get('success', False)), message
1381
1382    def schedule(
1383        self,
1384        schedule: str,
1385        name: Optional[str] = None,
1386        workers: Optional[int] = None,
1387        retries: int = 0,
1388        force: bool = False,
1389        executor_keys: Optional[str] = None,
1390        start: bool = True,
1391    ) -> 'mrsm.Job':
1392        """
1393        Create (and start) a background job that syncs the saved Pipeline on a schedule,
1394        like Manage → Schedule on the console's Pipelines page.
1395
1396        Parameters
1397        ----------
1398        schedule: str
1399            When to run, e.g. `'every 10 minutes'` or `'daily starting 00:30'`.
1400
1401        name: Optional[str], default None
1402            The job's name (default `pipeline-NAME`).
1403
1404        workers: Optional[int], default None
1405            Run up to this many pipes at once (`--workers`).
1406
1407        retries: int, default 0
1408            Retry a failed pipe this many times (`--retries`).
1409
1410        force: bool, default False
1411            Confirm rebuilds without asking (`--force`).
1412
1413        executor_keys: Optional[str], default None
1414            Where the job runs (`'local'`, `'systemd'` or an `api:` instance; default: as `Job`).
1415
1416        start: bool, default True
1417            Start the job now (a job that is already running is left running).
1418
1419        Returns
1420        -------
1421        The `mrsm.Job`. Raises `ValueError` for a Pipeline that is not saved, an invalid
1422        schedule, or a job of that name that runs a different command, and `RuntimeError` if
1423        the job does not start.
1424        """
1425        from meerschaum.utils.schedule import parse_schedule
1426        from meerschaum.core.Pipeline._saved import job_arguments
1427        if not self.exists():
1428            raise ValueError(
1429                "Save the Pipeline before scheduling it, e.g. `pipeline.save('daily')`."
1430            )
1431        sysargs = job_arguments(
1432            self.name, self.instance, schedule, workers=workers, retries=retries, force=force,
1433        )
1434        parse_schedule(str(schedule))
1435        job_name = name or f'pipeline-{self.name}'
1436        job = mrsm.Job(job_name, executor_keys=executor_keys)
1437        if job.exists():
1438            if list(job.sysargs) != sysargs:
1439                raise ValueError(
1440                    f"A job named '{job_name}' already exists with a different command "
1441                    f"({' '.join(job.sysargs)}). Pass `name=` for a new job, or delete that "
1442                    "one first."
1443                )
1444        else:
1445            job = mrsm.Job(job_name, sysargs, executor_keys=executor_keys)
1446        if start and not job.is_running():
1447            success, message = job.start()
1448            if not success:
1449                raise RuntimeError(f"Job '{job_name}' did not start: {message}")
1450        return job
1451
1452    def get_runs(
1453        self,
1454        limit: int = 20,
1455        status: Optional[str] = None,
1456        all_runs: bool = False,
1457        since: Union[datetime, timedelta, str, None] = None,
1458    ) -> List[Dict[str, Any]]:
1459        """
1460        Recent runs on the instance, newest first.
1461
1462        Parameters
1463        ----------
1464        limit: int, default 20
1465            The most runs to return.
1466
1467        status: Optional[str], default None
1468            Only `'running'`, `'succeeded'` or `'failed'` runs.
1469
1470        all_runs: bool, default False
1471            By default a named Pipeline returns its own runs (those of the saved Pipeline, or
1472            whose run IDs carry its name), and an unnamed one the runs which targeted or
1473            contained one of its member pipes. If `True` (or the selection is a tags / keys
1474            filter), return every run on the instance.
1475
1476        since: Union[datetime, timedelta, str, None], default None
1477            Only runs started at or after this time: a `datetime` (naive means UTC), a
1478            `timedelta` (that long ago, e.g. `timedelta(hours=1)`) or an ISO 8601 string.
1479
1480        Returns
1481        -------
1482        Run summaries: `run` (the stored row), `status`, `duration_seconds` and `counts`
1483        (nodes by status).
1484        """
1485        request = self._instance_request()
1486        request['limit'] = limit
1487        if status is not None:
1488            request['status'] = status
1489        since = since_text(since)
1490        if since is not None:
1491            request['since'] = since
1492        if all_runs:
1493            return call('engine_runs', request)
1494        if self.name:
1495            ### The runs of the saved Pipeline (and of combined selections that included it), or
1496            ### those whose run IDs carry the name.
1497            request['run_name'] = self.name
1498            return call('engine_runs', request)
1499        pipe_targets = [t for t in self._targets if isinstance(t, mrsm.Pipe)]
1500        if len(pipe_targets) != 1 or len(self._targets) != 1:
1501            runs = call('engine_runs', request)
1502            if pipe_targets:
1503                refs = {_pipe_ref(pipe) for pipe in pipe_targets}
1504                runs = [
1505                    run for run in runs
1506                    if refs.intersection(run['run'].get('targets', []) or [])
1507                ]
1508            return runs
1509        request['pipe'] = _pipe_ref(pipe_targets[0])
1510        return call('engine_runs', request)
1511
1512    def get_run(self, run_id: str) -> Optional[Dict[str, Any]]:
1513        """
1514        One run with its node rows and the planned-vs-actual Gantt bars
1515        (`None` if the run is unknown).
1516        """
1517        request = self._instance_request()
1518        request['run_id'] = run_id
1519        return call('engine_run_detail', request)
1520
1521    def get_events(self, run_id: str, after_seq: Optional[int] = None) -> List[Dict[str, Any]]:
1522        """The persisted events of a run, in order (see `RunResult.events()`)."""
1523        request = self._instance_request()
1524        request['run_id'] = run_id
1525        if after_seq is not None:
1526            request['after_seq'] = after_seq
1527        return _decode_events(call('engine_run_events', request))
1528
1529    def __repr__(self) -> str:
1530        if self.name:
1531            import sys
1532            from meerschaum.config import get_config
1533            if hasattr(sys, 'ps1'):
1534                from meerschaum.utils.formatting._pipes import pipeline_repr
1535                return pipeline_repr(self.name, str(self.instance))
1536            suffix = f', instance={str(self.instance)!r}' if str(self.instance) != get_config('meerschaum', 'instance') else ''
1537            return f'Pipeline({self.name!r}{suffix})'
1538        if self._definitions:
1539            parts = [repr(self._sources)]
1540        else:
1541            parts = [repr(self._targets)] if self._targets else [repr(self._filters)]
1542        if self.parents:
1543            parts.append(f"parents={self.parents!r}")
1544        if self.children:
1545            parts.append(f"children={self.children!r}")
1546        return f"Pipeline({', '.join(parts)})"

A selection over the pipe graph: member pipes plus traversal depths (design §5.2, D4).

Parameters

  • pipes (Union[str, mrsm.Pipe, Iterable[...], None], default None): What to select:

    • a str without | is the name of a saved Pipeline, loaded from instance (mrsm.Pipeline('daily'); ValueError if it does not exist);
    • a str with | is a pipe reference, 'instance|connector|metric|location';
    • a Pipe, or a (connector_keys, metric_key[, location_key]) tuple, is one member;
    • a list may mix all of these: the members of every named Pipeline plus the given pipes (the dictionary mrsm.get_pipes() returns works too).
  • parents (Union[bool, int, None], default None): True includes every ancestor of the members; an int includes that many generations. For a saved Pipeline, None keeps its saved setting and False turns traversal off.
  • children (Union[bool, int, None], default None): True includes every descendant of the members; an int that many generations. For a saved Pipeline, None keeps its saved setting and False turns traversal off.
  • tags, connector_keys, metric_keys, location_keys (Union[str, List[str], None], default None): Select the members by the same filters as -t, -c, -m and -l (negate a value with a leading _). Used when pipes is not provided.
  • instance (Union[str, InstanceConnector, None], default None): The instance that keeps the engine state and the saved Pipelines (default: the member pipes' instance, else the configured one). Saved Pipelines are loaded from here.
  • name (Optional[str], default None): The name to save this selection under (save()), also used in its run IDs. Without pipes or filters, Pipeline(name='daily') loads the saved Pipeline.
  • graph (Optional[bool], default None): Whether the run respects the dependencies between the members even without traversal (a child member waits for its parent member and is capped at its watermark). A saved Pipeline keeps its graph setting (default True), like mrsm sync pipelines NAME; an unsaved selection without parents / children syncs each member on its own, like mrsm sync pipes.
Examples
>>> pipeline = mrsm.Pipeline(pipe, parents=True)
>>> plan = pipeline.plan()
>>> run = pipeline.sync(workers=4)
>>> run.success, run.run_id, run.inserted
>>> pipeline.save('daily')
>>> mrsm.Pipeline('daily').sync()
Pipeline( pipes: str | Pipe | Iterable[Any] | None = None, parents: bool | int | None = None, children: bool | int | None = None, tags: str | List[str] | None = None, connector_keys: str | List[str] | None = None, metric_keys: str | List[str] | None = None, location_keys: str | List[str] | None = None, instance: str | Any | None = None, name: str | None = None, graph: bool | None = None) 
641    def __init__(
642        self,
643        pipes: Union[str, mrsm.Pipe, Iterable[Any], None] = None,
644        parents: Union[bool, int, None] = None,
645        children: Union[bool, int, None] = None,
646        tags: Union[str, List[str], None] = None,
647        connector_keys: Union[str, List[str], None] = None,
648        metric_keys: Union[str, List[str], None] = None,
649        location_keys: Union[str, List[str], None] = None,
650        instance: Union[str, Any, None] = None,
651        name: Optional[str] = None,
652        graph: Optional[bool] = None,
653    ):
654        if name is not None and not isinstance(name, str):
655            raise TypeError(f"A Pipeline name must be a string (got {name!r}).")
656        names, targets = _split_selection(pipes)
657        filters = {
658            'tags': tags,
659            'connector_keys': connector_keys,
660            'metric_keys': metric_keys,
661            'location_keys': location_keys,
662        }
663        filters = {k: ([v] if isinstance(v, str) else v) for k, v in filters.items() if v}
664        if not names and not targets and not filters:
665            if not name:
666                raise ValueError(
667                    "A Pipeline needs member pipes, a tags / keys filter or the name of a saved "
668                    "Pipeline, e.g. `mrsm.Pipeline(pipe)`, `mrsm.Pipeline(tags=['daily'])` or "
669                    "`mrsm.Pipeline('daily')`."
670                )
671            names = [name]
672        if names and filters:
673            raise ValueError(
674                "A saved Pipeline cannot be combined with a tags / keys filter: "
675                "pass the extra pipes themselves."
676            )
677        definitions = []
678        registry = None
679        if names:
680            registry = _registry_of(instance)
681            seen = set()
682            for saved_name in names:
683                if saved_name in seen:
684                    continue
685                seen.add(saved_name)
686                definitions.append(_read_definition(saved_name, registry))
687        self._configure(
688            definitions,
689            targets,
690            filters,
691            parents=parents,
692            children=children,
693            instance=(registry if registry is not None else instance),
694            name=name,
695            graph=graph,
696        )
@classmethod
def load(cls, name: str, instance=None) -> Pipeline: 
776    @classmethod
777    def load(cls, name: str, instance=None) -> 'Pipeline':
778        """
779        Load the saved Pipeline `name` from `instance` (default: the configured instance).
780        The same as `mrsm.Pipeline(name, instance=instance)`.
781        """
782        return cls(name, instance=instance)

Load the saved Pipeline name from instance (default: the configured instance). The same as mrsm.Pipeline(name, instance=instance).

definition: Dict[str, Any] | None 
784    @property
785    def definition(self) -> Optional[Dict[str, Any]]:
786        """
787        The saved definition this Pipeline was loaded from or saved as (`name`, `registry`,
788        `id`, `revision`, `targets`, `parents`, `children`, `description`, `defaults`, ...),
789        or `None` for an unsaved selection (or one combined from several saved Pipelines).
790        """
791        return self._definitions[0] if len(self._definitions) == 1 else None

The saved definition this Pipeline was loaded from or saved as (name, registry, id, revision, targets, parents, children, description, defaults, ...), or None for an unsaved selection (or one combined from several saved Pipelines).

graph: bool 
818    @property
819    def graph(self) -> bool:
820        """
821        Whether runs respect the dependencies between members even without traversal
822        (see the `graph` argument).
823        """
824        if self._graph is not None:
825            return self._graph
826        if not self._definitions:
827            return False
828        return _common([d.get('graph', True) for d in self._definitions],
829                       'graph', [d['name'] for d in self._definitions])

Whether runs respect the dependencies between members even without traversal (see the graph argument).

members: List[Any] 
835    @property
836    def members(self) -> List[Any]:
837        """
838        The member pipes (or keys) the selection starts from, before traversal
839        (empty for a tags / keys filter). Also available as `targets`.
840
841        Assign a new list (`Pipe` objects, key tuples or `pipe_ref` strings) to change the
842        selection, then `save()` to keep the change:
843
844        ```python
845        daily = mrsm.Pipeline('daily')
846        daily.members = daily.members + [mrsm.Pipe('sql:main', 'weekly')]
847        daily.save()
848        ```
849        """
850        return list(self._targets)

The member pipes (or keys) the selection starts from, before traversal (empty for a tags / keys filter). Also available as targets.

Assign a new list (Pipe objects, key tuples or pipe_ref strings) to change the selection, then save() to keep the change:

daily = mrsm.Pipeline('daily')
daily.members = daily.members + [mrsm.Pipe('sql:main', 'weekly')]
daily.save()
targets: List[Any] 
878    @property
879    def targets(self) -> List[Any]:
880        """The member pipes (or keys) the selection starts from (the same as `members`)."""
881        return list(self._targets)

The member pipes (or keys) the selection starts from (the same as members).

pipes: List[Pipe] 
887    @property
888    def pipes(self) -> List[mrsm.Pipe]:
889        """
890        Every pipe the selection resolves to (members plus the traversed parents and children),
891        in dependency order. This plans the selection, registering nothing: member pipes that
892        are not registered yet come last (their dependencies are unknown until they are).
893        """
894        plan = self.plan()
895        planned = plan.pipes
896        return planned + [pipe for pipe in plan.unregistered if pipe not in planned]

Every pipe the selection resolves to (members plus the traversed parents and children), in dependency order. This plans the selection, registering nothing: member pipes that are not registered yet come last (their dependencies are unknown until they are).

def plan( self, begin: Any = <object object>, end: Any = <object object>, expected: bool = False, **kwargs: Any) -> Plan: 
 967    def plan(
 968        self,
 969        begin: Any = _UNSET_BOUND,
 970        end: Any = _UNSET_BOUND,
 971        expected: bool = False,
 972        **kwargs: Any
 973    ) -> Plan:
 974        """
 975        Plan the run without moving any data (and without registering anything).
 976
 977        Parameters
 978        ----------
 979        begin, end: Union[datetime, int, str, None], default None
 980            Optionally bound the sync window. `None` and `''` leave it to the planner
 981            (a root syncs from its sync time minus the backtrack).
 982
 983        expected: bool, default False
 984            If `False`, every node is planned against the state as it is now: a node whose
 985            parents run first is provisional (`*`) and is planned again by the executor once
 986            they finish. If `True`, return what `mrsm plan pipes` prints: each provisional node
 987            re-planned as if every node that runs before it synced new rows after its
 988            watermark, so its window starts where the executor will start it (often the start
 989            of a parent's chunk-grid cell). Only the start is predicted; the executor caps the
 990            end at the parents' new watermarks.
 991
 992        Returns
 993        -------
 994        A `Plan` (`str(plan)` is the table; `plan.nodes` is the data). Member pipes that are
 995        not registered yet are in `plan.unregistered` and named in `plan.warnings`.
 996        """
 997        options = {key: value for key, value in {'begin': begin, 'end': end, **kwargs}.items()
 998                   if value is not _UNSET_BOUND}
 999        unregistered = self._unregistered()
1000        request = self._request(
1001            targets=(
1002                [target for target in self._targets if target not in unregistered]
1003                if unregistered
1004                else None
1005            ),
1006            **options
1007        )
1008        if expected:
1009            request['expected'] = True
1010        data = call('engine_plan', request)
1011        return Plan(
1012            data,
1013            pipeline=self,
1014            options={
1015                key: value
1016                for key, value in options.items()
1017                if value not in (None, '')
1018            },
1019            unregistered=unregistered,
1020        )

Plan the run without moving any data (and without registering anything).

Parameters
  • begin, end (Union[datetime, int, str, None], default None): Optionally bound the sync window. None and '' leave it to the planner (a root syncs from its sync time minus the backtrack).
  • expected (bool, default False): If False, every node is planned against the state as it is now: a node whose parents run first is provisional (*) and is planned again by the executor once they finish. If True, return what mrsm plan pipes prints: each provisional node re-planned as if every node that runs before it synced new rows after its watermark, so its window starts where the executor will start it (often the start of a parent's chunk-grid cell). Only the start is predicted; the executor caps the end at the parents' new watermarks.
Returns
  • A Plan (str(plan) is the table; plan.nodes is the data). Member pipes that are
  • not registered yet are in plan.unregistered and named in plan.warnings.
def sync( self, begin: Any = <object object>, end: Any = <object object>, **kwargs: Any) -> RunResult: 
1022    def sync(self, begin: Any = _UNSET_BOUND, end: Any = _UNSET_BOUND, **kwargs: Any) -> RunResult:
1023        """
1024        Plan and run the selection, registering member pipes that are not registered yet
1025        (a saved Pipeline's members must already be registered).
1026
1027        Parameters
1028        ----------
1029        begin, end: Union[datetime, int, str, None]
1030            Bound the sync window of the root nodes (other nodes are clamped to it).
1031            `None` and `''` leave the window to the planner: a root syncs from its sync time
1032            minus the backtrack, as a bare `pipe.sync()` does by default. (A bare
1033            `pipe.sync(begin=None)` means "no lower bound"; in an engine run it does not.)
1034
1035        workers: Optional[int], default None
1036            Run up to this many nodes at once (default `system.engine.workers`).
1037
1038        force, retries, min_seconds, check_existing, enforce_dtypes, chunksize, params, yes:
1039            The `Pipe.sync()` options, applied to every node. `yes` (or `force`) confirms a
1040            `rebuild`; without it a node whose definition changed is left `rebuild_pending`.
1041
1042        resume: Optional[str], default None
1043            Re-plan only the nodes that failed or never ran in this earlier run ID.
1044
1045        on_event: Optional[Callable[[Dict[str, Any]], Any]], default None
1046            Called with every event of the run as it happens (`run_started`, `node_planned`,
1047            `node_started`, `chunk_started`, `chunk_finished`, `node_finished`, `node_failed`,
1048            `node_skipped`, `watermark_advanced`, ..., `run_finished`), each a `dict` with
1049            `event`, `run_id`, `seq`, `ts` (an aware UTC `datetime`), `pipe_ref` and the
1050            event's own fields. It runs on this thread; the sync work runs on the engine's
1051            threads without the GIL. If it raises an `Exception`, the run continues, the error
1052            is warned about once and kept in `RunResult.callback_error`, and no further events
1053            are delivered. `KeyboardInterrupt` cancels the run.
1054
1055        Returns
1056        -------
1057        A `RunResult`: a `SuccessTuple` with `run_id`, `plan`, `outcomes`, `events()`, the
1058        totals `inserted`, `updated`, `upserted` and `deleted` over every node, and `duration`
1059        (the run's wall-clock seconds).
1060        """
1061        on_event = kwargs.pop('on_event', None)
1062        failure = self._register_targets()
1063        if failure is not None:
1064            return RunResult(*failure)
1065        options = {key: value for key, value in {'begin': begin, 'end': end, **kwargs}.items()
1066                   if value is not _UNSET_BOUND}
1067        try:
1068            request = self._request(**options)
1069        except ValueError as e:
1070            return RunResult(False, str(e))
1071        callback = EventCallback(on_event) if on_event is not None else None
1072        if callback is None:
1073            data = call('engine_run', request)
1074        else:
1075            data = call('engine_run', request, on_event=callback)
1076            if callback.warning is not None:
1077                data['warnings'] = list(data.get('warnings', None) or []) + [callback.warning]
1078        run = RunResult(data['success'], data['message'], data, pipeline=self)
1079        run._instance_request = self._instance_request()
1080        if callback is not None:
1081            run.callback_error = callback.error
1082        return run

Plan and run the selection, registering member pipes that are not registered yet (a saved Pipeline's members must already be registered).

Parameters
  • begin, end (Union[datetime, int, str, None]): Bound the sync window of the root nodes (other nodes are clamped to it). None and '' leave the window to the planner: a root syncs from its sync time minus the backtrack, as a bare pipe.sync() does by default. (A bare pipe.sync(begin=None) means "no lower bound"; in an engine run it does not.)
  • workers (Optional[int], default None): Run up to this many nodes at once (default system.engine.workers).
  • force, retries, min_seconds, check_existing, enforce_dtypes, chunksize, params, yes:: The Pipe.sync() options, applied to every node. yes (or force) confirms a rebuild; without it a node whose definition changed is left rebuild_pending.
  • resume (Optional[str], default None): Re-plan only the nodes that failed or never ran in this earlier run ID.
  • on_event (Optional[Callable[[Dict[str, Any]], Any]], default None): Called with every event of the run as it happens (run_started, node_planned, node_started, chunk_started, chunk_finished, node_finished, node_failed, node_skipped, watermark_advanced, ..., run_finished), each a dict with event, run_id, seq, ts (an aware UTC datetime), pipe_ref and the event's own fields. It runs on this thread; the sync work runs on the engine's threads without the GIL. If it raises an Exception, the run continues, the error is warned about once and kept in RunResult.callback_error, and no further events are delivered. KeyboardInterrupt cancels the run.
Returns

  • A RunResult (a SuccessTuple with run_id, plan, outcomes, events(), the):

  • totals inserted, updated, upserted and deleted over every node, and duration

  • (the run's wall-clock seconds).
def run( self, begin: Any = <object object>, end: Any = <object object>, **kwargs: Any) -> RunResult: 
1084    def run(self, begin: Any = _UNSET_BOUND, end: Any = _UNSET_BOUND, **kwargs: Any) -> RunResult:
1085        """An alias of `Pipeline.sync()`."""
1086        return self.sync(begin=begin, end=end, **kwargs)

An alias of Pipeline.sync().

def save( self, name: str | None = None, overwrite: bool = False, description: str | None = None, defaults: Dict[str, Any] | None = None) -> Tuple[bool, str]: 
1126    def save(
1127        self,
1128        name: Optional[str] = None,
1129        overwrite: bool = False,
1130        description: Optional[str] = None,
1131        defaults: Optional[Dict[str, Any]] = None,
1132    ) -> SuccessTuple:
1133        """
1134        Save this selection's members, traversal and execution mode under `name`
1135        (default: `self.name`). Saving never changes the members' tags or registrations.
1136
1137        Parameters
1138        ----------
1139        name: Optional[str], default None
1140            The name to save under (letters, numbers, dots, underscores and hyphens).
1141
1142        overwrite: bool, default False
1143            A different Pipeline already saved under `name` is only replaced with
1144            `overwrite=True` (its description, defaults and layout are kept unless passed).
1145            A Pipeline loaded from `name` saves its changes without it, unless someone else
1146            saved a newer revision since it was loaded (load it again). Saving the same
1147            members, traversal and execution mode again is a no-op success.
1148
1149        description: Optional[str], default None
1150            Set the saved description.
1151
1152        defaults: Optional[Dict[str, Any]], default None
1153            Set the saved execution defaults (`begin`, `end`, `workers`, `retries`,
1154            `min_seconds`, `chunksize`, `check_existing`, `enforce_dtypes`, `force`, `params`,
1155            `yes`), used by every run of the saved Pipeline unless a run passes its own.
1156
1157        Returns
1158        -------
1159        A `SuccessTuple`. On success, `self.name` and `self.definition` are the saved ones.
1160        """
1161        from meerschaum.core.Pipeline._saved import (
1162            _read, _refs, registration, save_pipeline, validate_name,
1163        )
1164        name = name or self.name
1165        if not name:
1166            return False, "Name the Pipeline to save it, e.g. `pipeline.save('daily')`."
1167        try:
1168            validate_name(name)
1169        except ValueError as e:
1170            return False, f"Cannot save Pipeline {name!r}: {e}"
1171
1172        registry = self.instance
1173        try:
1174            members = self._members_to_save()
1175            graph = True if self._graph is None and not self._definitions else self.graph
1176            registration_pipe = registration(name, registry)
1177            registry_keys = registration_pipe.instance_keys
1178            exists = registration_pipe.get_id() is not None
1179            source = self.definition
1180            editing = (
1181                exists
1182                and source is not None
1183                and source.get('name', None) == name
1184                and str(source.get('registry', None)) == str(registry_keys)
1185            )
1186            expected_revision = None
1187            if exists and not overwrite:
1188                ### Saving what is already saved is a no-op (no new revision for other editors
1189                ### to trip over). Otherwise a Pipeline loaded from `name` saves its changes
1190                ### against the revision it loaded, and replacing a different one needs
1191                ### `overwrite`.
1192                current = _read(name, registry)
1193                same_selection = (
1194                    sorted(current.get('targets', None) or []) == _refs(members, registry_keys)
1195                    and get_depth(current.get('parents', None)) == self.parents
1196                    and get_depth(current.get('children', None)) == self.children
1197                    and current.get('graph', True) == graph
1198                )
1199                unchanged = (
1200                    (description is None or description == current.get('description', ''))
1201                    and (defaults is None or defaults == current.get('defaults', {}))
1202                )
1203                if same_selection and unchanged:
1204                    self._bind([current])
1205                    self.name = name
1206                    return True, (
1207                        f"Pipeline '{name}' is already saved on '{registry_keys}' "
1208                        "with these members."
1209                    )
1210                if not editing and not same_selection:
1211                    return False, (
1212                        f"A different Pipeline '{name}' is already saved on '{registry_keys}'. "
1213                        f"Load it with `mrsm.Pipeline({name!r})` to change it, or pass "
1214                        "`overwrite=True` to replace it."
1215                    )
1216                expected_revision = source['revision'] if editing else current['revision']
1217
1218            if exists:
1219                definition = save_pipeline(
1220                    name, members, registry, self.parents, self.children,
1221                    expected_revision=expected_revision,
1222                    description=description,
1223                    defaults=defaults,
1224                    graph=graph,
1225                )
1226            else:
1227                carried = self._settings_to_carry(with_defaults=(defaults is None))
1228                definition = save_pipeline(
1229                    name, members, registry, self.parents, self.children,
1230                    description=(description if description is not None else carried.get('description', None)),
1231                    defaults=(defaults if defaults is not None else carried.get('defaults', None)),
1232                    layout=carried.get('layout', None),
1233                    create=True,
1234                    graph=graph,
1235                )
1236        except Exception as e:
1237            return False, f"Failed to save Pipeline '{name}' on '{registry}': {e}"
1238
1239        self._bind([definition])
1240        self.name = name
1241        num_members = len(definition.get('targets', None) or [])
1242        try:
1243            num_unregistered = len(self._unregistered())
1244        except Exception:
1245            num_unregistered = 0
1246        note = (
1247            f" {num_unregistered} of them {'is' if num_unregistered == 1 else 'are'} not "
1248            "registered yet: `sync()` registers them."
1249            if num_unregistered
1250            else ''
1251        )
1252        return True, (
1253            f"Saved Pipeline '{name}' on '{definition['registry']}' "
1254            f"({num_members} member{'s' if num_members != 1 else ''}, "
1255            f"revision {definition['revision']}).{note}"
1256        )

Save this selection's members, traversal and execution mode under name (default: self.name). Saving never changes the members' tags or registrations.

Parameters
  • name (Optional[str], default None): The name to save under (letters, numbers, dots, underscores and hyphens).
  • overwrite (bool, default False): A different Pipeline already saved under name is only replaced with overwrite=True (its description, defaults and layout are kept unless passed). A Pipeline loaded from name saves its changes without it, unless someone else saved a newer revision since it was loaded (load it again). Saving the same members, traversal and execution mode again is a no-op success.
  • description (Optional[str], default None): Set the saved description.
  • defaults (Optional[Dict[str, Any]], default None): Set the saved execution defaults (begin, end, workers, retries, min_seconds, chunksize, check_existing, enforce_dtypes, force, params, yes), used by every run of the saved Pipeline unless a run passes its own.
Returns
  • A SuccessTuple. On success, self.name and self.definition are the saved ones.
def exists(self) -> bool: 
1258    def exists(self) -> bool:
1259        """Whether a Pipeline is saved under this Pipeline's `name` on its instance."""
1260        if not self.name:
1261            return False
1262        from meerschaum.core.Pipeline._saved import registration
1263        try:
1264            return registration(self.name, self.instance).get_id() is not None
1265        except ValueError:
1266            return False

Whether a Pipeline is saved under this Pipeline's name on its instance.

def delete(self) -> Tuple[bool, str]: 
1268    def delete(self) -> SuccessTuple:
1269        """
1270        Delete the saved Pipeline (its definition and the membership tags it owns).
1271        The member pipes and their data are kept, and this object stays usable as an unsaved
1272        selection. Scheduled jobs that sync the Pipeline must be removed first.
1273        """
1274        from meerschaum.core.Pipeline._saved import delete_pipeline
1275        if not self.name:
1276            return False, "This Pipeline has no name, so it is not saved."
1277        try:
1278            success, message = delete_pipeline(self.name, self.instance)
1279        except Exception as e:
1280            return False, f"Failed to delete Pipeline '{self.name}': {e}"
1281        if not success:
1282            return success, message
1283        self._bind([])
1284        return True, (
1285            f"Deleted Pipeline '{self.name}' from '{self.instance}' "
1286            "(its member pipes and their data are kept)."
1287        )

Delete the saved Pipeline (its definition and the membership tags it owns). The member pipes and their data are kept, and this object stays usable as an unsaved selection. Scheduled jobs that sync the Pipeline must be removed first.

def rename(self, new_name: str) -> Tuple[bool, str]: 
1289    def rename(self, new_name: str) -> SuccessTuple:
1290        """
1291        Rename the saved Pipeline (its identity and run history are kept).
1292        Scheduled jobs that sync the Pipeline must be removed first.
1293        """
1294        from meerschaum.core.Pipeline._saved import rename_pipeline
1295        if not self.exists():
1296            return False, f"Pipeline '{self.name}' is not saved on '{self.instance}'."
1297        definition = self.definition
1298        expected_revision = (
1299            definition['revision']
1300            if definition is not None and definition.get('name', None) == self.name
1301            else None
1302        )
1303        old_name = self.name
1304        try:
1305            renamed = rename_pipeline(
1306                old_name, new_name, self.instance, expected_revision=expected_revision,
1307            )
1308        except Exception as e:
1309            return False, f"Failed to rename Pipeline '{old_name}': {e}"
1310        self._bind([renamed])
1311        self.name = new_name
1312        if old_name == new_name:
1313            return True, f"Pipeline '{old_name}' already has that name."
1314        return True, f"Renamed Pipeline '{old_name}' to '{new_name}' on '{self.instance}'."

Rename the saved Pipeline (its identity and run history are kept). Scheduled jobs that sync the Pipeline must be removed first.

def copy( self, new_name: str, instance: str | Any | None = None) -> Tuple[bool, str]: 
1316    def copy(self, new_name: str, instance: Union[str, Any, None] = None) -> SuccessTuple:
1317        """
1318        Save a copy of this Pipeline as `new_name`, on `instance` (default: this Pipeline's
1319        instance). The copy has the same members (which keep their own instances), traversal,
1320        description and defaults, and a new identity; this object is unchanged.
1321        """
1322        from meerschaum.core.Pipeline._saved import save_pipeline, validate_name
1323        try:
1324            validate_name(new_name)
1325        except ValueError as e:
1326            return False, f"Cannot copy to {new_name!r}: {e}"
1327        registry = _registry_of(instance) if instance is not None else self.instance
1328        try:
1329            carried = self._settings_to_carry()
1330            definition = save_pipeline(
1331                new_name, self._members_to_save(), registry, self.parents, self.children,
1332                description=carried.get('description', None),
1333                defaults=carried.get('defaults', None),
1334                layout=carried.get('layout', None),
1335                create=True,
1336            )
1337        except Exception as e:
1338            return False, f"Failed to copy to Pipeline '{new_name}' on '{registry}': {e}"
1339        source = f"Pipeline '{self.name}'" if self.name else 'this selection'
1340        return True, f"Copied {source} to Pipeline '{new_name}' on '{definition['registry']}'."

Save a copy of this Pipeline as new_name, on instance (default: this Pipeline's instance). The copy has the same members (which keep their own instances), traversal, description and defaults, and a new identity; this object is unchanged.

def analyze(self) -> Dict[str, Any]: 
1342    def analyze(self) -> Dict[str, Any]:
1343        """
1344        Resolve the saved Pipeline's members and traversal and preview the membership tags
1345        (`pipeline-NAME`) it would add or remove, without changing anything (like
1346        `mrsm analyze pipelines NAME`). Apply the proposal with `apply_analysis()`.
1347
1348        Returns
1349        -------
1350        The proposal: `resolved` (each pipe and why it is included), `changes` (tag edits),
1351        `shared` (pipes other Pipelines also use), `errors` and a `fingerprint`.
1352        """
1353        from meerschaum.core.Pipeline._saved import analyze_pipeline
1354        if not self.name:
1355            raise ValueError("Save the Pipeline before analyzing it, e.g. `pipeline.save('daily')`.")
1356        return analyze_pipeline(self.name, self.instance)

Resolve the saved Pipeline's members and traversal and preview the membership tags (pipeline-NAME) it would add or remove, without changing anything (like mrsm analyze pipelines NAME). Apply the proposal with apply_analysis().

Returns

  • The proposal (resolved (each pipe and why it is included), changes (tag edits),):

  • shared (pipes other Pipelines also use), errors and a fingerprint.

def apply_analysis(self, proposal: Dict[str, Any] | None = None) -> Tuple[bool, str]: 
1358    def apply_analysis(self, proposal: Optional[Dict[str, Any]] = None) -> SuccessTuple:
1359        """
1360        Apply the membership tag changes of a proposal from `analyze()` (without one, analyze
1361        now and apply that, like `mrsm analyze pipelines NAME --yes`). A proposal whose inputs
1362        changed since it was made is refused: analyze again and review the new one.
1363        """
1364        from meerschaum.core.Pipeline._saved import _read, apply_pipeline_analysis
1365        if not self.name:
1366            return False, "This Pipeline has no name, so it is not saved."
1367        try:
1368            proposal = proposal if proposal is not None else self.analyze()
1369            result = apply_pipeline_analysis(proposal, self.instance)
1370        except Exception as e:
1371            return False, f"Failed to apply the analysis of Pipeline '{self.name}': {e}"
1372        message = result.get('message', '')
1373        failures = [row for row in result.get('results', None) or [] if not row.get('success', True)]
1374        if failures:
1375            message += ''.join(f"\n{row['pipe_ref']}: {row['message']}" for row in failures)
1376        try:
1377            self._bind([_read(self.name, self.instance)])
1378        except Exception:
1379            pass
1380        return bool(result.get('success', False)), message

Apply the membership tag changes of a proposal from analyze() (without one, analyze now and apply that, like mrsm analyze pipelines NAME --yes). A proposal whose inputs changed since it was made is refused: analyze again and review the new one.

def schedule( self, schedule: str, name: str | None = None, workers: int | None = None, retries: int = 0, force: bool = False, executor_keys: str | None = None, start: bool = True) -> Job: 
1382    def schedule(
1383        self,
1384        schedule: str,
1385        name: Optional[str] = None,
1386        workers: Optional[int] = None,
1387        retries: int = 0,
1388        force: bool = False,
1389        executor_keys: Optional[str] = None,
1390        start: bool = True,
1391    ) -> 'mrsm.Job':
1392        """
1393        Create (and start) a background job that syncs the saved Pipeline on a schedule,
1394        like Manage → Schedule on the console's Pipelines page.
1395
1396        Parameters
1397        ----------
1398        schedule: str
1399            When to run, e.g. `'every 10 minutes'` or `'daily starting 00:30'`.
1400
1401        name: Optional[str], default None
1402            The job's name (default `pipeline-NAME`).
1403
1404        workers: Optional[int], default None
1405            Run up to this many pipes at once (`--workers`).
1406
1407        retries: int, default 0
1408            Retry a failed pipe this many times (`--retries`).
1409
1410        force: bool, default False
1411            Confirm rebuilds without asking (`--force`).
1412
1413        executor_keys: Optional[str], default None
1414            Where the job runs (`'local'`, `'systemd'` or an `api:` instance; default: as `Job`).
1415
1416        start: bool, default True
1417            Start the job now (a job that is already running is left running).
1418
1419        Returns
1420        -------
1421        The `mrsm.Job`. Raises `ValueError` for a Pipeline that is not saved, an invalid
1422        schedule, or a job of that name that runs a different command, and `RuntimeError` if
1423        the job does not start.
1424        """
1425        from meerschaum.utils.schedule import parse_schedule
1426        from meerschaum.core.Pipeline._saved import job_arguments
1427        if not self.exists():
1428            raise ValueError(
1429                "Save the Pipeline before scheduling it, e.g. `pipeline.save('daily')`."
1430            )
1431        sysargs = job_arguments(
1432            self.name, self.instance, schedule, workers=workers, retries=retries, force=force,
1433        )
1434        parse_schedule(str(schedule))
1435        job_name = name or f'pipeline-{self.name}'
1436        job = mrsm.Job(job_name, executor_keys=executor_keys)
1437        if job.exists():
1438            if list(job.sysargs) != sysargs:
1439                raise ValueError(
1440                    f"A job named '{job_name}' already exists with a different command "
1441                    f"({' '.join(job.sysargs)}). Pass `name=` for a new job, or delete that "
1442                    "one first."
1443                )
1444        else:
1445            job = mrsm.Job(job_name, sysargs, executor_keys=executor_keys)
1446        if start and not job.is_running():
1447            success, message = job.start()
1448            if not success:
1449                raise RuntimeError(f"Job '{job_name}' did not start: {message}")
1450        return job

Create (and start) a background job that syncs the saved Pipeline on a schedule, like Manage → Schedule on the console's Pipelines page.

Parameters
  • schedule (str): When to run, e.g. 'every 10 minutes' or 'daily starting 00:30'.
  • name (Optional[str], default None): The job's name (default pipeline-NAME).
  • workers (Optional[int], default None): Run up to this many pipes at once (--workers).
  • retries (int, default 0): Retry a failed pipe this many times (--retries).
  • force (bool, default False): Confirm rebuilds without asking (--force).
  • executor_keys (Optional[str], default None): Where the job runs ('local', 'systemd' or an api: instance; default: as Job).
  • start (bool, default True): Start the job now (a job that is already running is left running).
Returns
  • The mrsm.Job. Raises ValueError for a Pipeline that is not saved, an invalid
  • schedule, or a job of that name that runs a different command, and RuntimeError if
  • the job does not start.
def get_runs( self, limit: int = 20, status: str | None = None, all_runs: bool = False, since: datetime.datetime | datetime.timedelta | str | None = None) -> List[Dict[str, Any]]: 
1452    def get_runs(
1453        self,
1454        limit: int = 20,
1455        status: Optional[str] = None,
1456        all_runs: bool = False,
1457        since: Union[datetime, timedelta, str, None] = None,
1458    ) -> List[Dict[str, Any]]:
1459        """
1460        Recent runs on the instance, newest first.
1461
1462        Parameters
1463        ----------
1464        limit: int, default 20
1465            The most runs to return.
1466
1467        status: Optional[str], default None
1468            Only `'running'`, `'succeeded'` or `'failed'` runs.
1469
1470        all_runs: bool, default False
1471            By default a named Pipeline returns its own runs (those of the saved Pipeline, or
1472            whose run IDs carry its name), and an unnamed one the runs which targeted or
1473            contained one of its member pipes. If `True` (or the selection is a tags / keys
1474            filter), return every run on the instance.
1475
1476        since: Union[datetime, timedelta, str, None], default None
1477            Only runs started at or after this time: a `datetime` (naive means UTC), a
1478            `timedelta` (that long ago, e.g. `timedelta(hours=1)`) or an ISO 8601 string.
1479
1480        Returns
1481        -------
1482        Run summaries: `run` (the stored row), `status`, `duration_seconds` and `counts`
1483        (nodes by status).
1484        """
1485        request = self._instance_request()
1486        request['limit'] = limit
1487        if status is not None:
1488            request['status'] = status
1489        since = since_text(since)
1490        if since is not None:
1491            request['since'] = since
1492        if all_runs:
1493            return call('engine_runs', request)
1494        if self.name:
1495            ### The runs of the saved Pipeline (and of combined selections that included it), or
1496            ### those whose run IDs carry the name.
1497            request['run_name'] = self.name
1498            return call('engine_runs', request)
1499        pipe_targets = [t for t in self._targets if isinstance(t, mrsm.Pipe)]
1500        if len(pipe_targets) != 1 or len(self._targets) != 1:
1501            runs = call('engine_runs', request)
1502            if pipe_targets:
1503                refs = {_pipe_ref(pipe) for pipe in pipe_targets}
1504                runs = [
1505                    run for run in runs
1506                    if refs.intersection(run['run'].get('targets', []) or [])
1507                ]
1508            return runs
1509        request['pipe'] = _pipe_ref(pipe_targets[0])
1510        return call('engine_runs', request)

Recent runs on the instance, newest first.

Parameters
  • limit (int, default 20): The most runs to return.
  • status (Optional[str], default None): Only 'running', 'succeeded' or 'failed' runs.
  • all_runs (bool, default False): By default a named Pipeline returns its own runs (those of the saved Pipeline, or whose run IDs carry its name), and an unnamed one the runs which targeted or contained one of its member pipes. If True (or the selection is a tags / keys filter), return every run on the instance.
  • since (Union[datetime, timedelta, str, None], default None): Only runs started at or after this time: a datetime (naive means UTC), a timedelta (that long ago, e.g. timedelta(hours=1)) or an ISO 8601 string.
Returns

  • Run summaries (run (the stored row), status, duration_seconds and counts):

  • (nodes by status).

def get_run(self, run_id: str) -> Dict[str, Any] | None: 
1512    def get_run(self, run_id: str) -> Optional[Dict[str, Any]]:
1513        """
1514        One run with its node rows and the planned-vs-actual Gantt bars
1515        (`None` if the run is unknown).
1516        """
1517        request = self._instance_request()
1518        request['run_id'] = run_id
1519        return call('engine_run_detail', request)

One run with its node rows and the planned-vs-actual Gantt bars (None if the run is unknown).

def get_events(self, run_id: str, after_seq: int | None = None) -> List[Dict[str, Any]]: 
1521    def get_events(self, run_id: str, after_seq: Optional[int] = None) -> List[Dict[str, Any]]:
1522        """The persisted events of a run, in order (see `RunResult.events()`)."""
1523        request = self._instance_request()
1524        request['run_id'] = run_id
1525        if after_seq is not None:
1526            request['after_seq'] = after_seq
1527        return _decode_events(call('engine_run_events', request))

The persisted events of a run, in order (see RunResult.events()).

class Plan: 
178class Plan:
179    """
180    A dry run: what the engine would do, with no data movement.
181
182    `str(plan)` is the plan table (`node | action | bounds | chunks | reason`, `*` marks a
183    provisional plan).
184
185    Attributes
186    ----------
187    nodes: List[PlannedNode]
188        The nodes in dependency order.
189
190    edges: List[Tuple[str, str]]
191        The `(parent, child)` pairs inside the selection.
192
193    flat: bool
194        `True` for a selection with no traversal and no dependencies between its members:
195        every node syncs on its own, like `mrsm sync pipes`.
196
197    warnings: List[str]
198        Graph and selection warnings (including one per member pipe that is not registered).
199
200    unregistered: List[mrsm.Pipe]
201        Member pipes that are not registered on their instance, so the engine cannot plan
202        them yet. `Pipeline.sync()` registers them first (except for a saved Pipeline, whose
203        members must already be registered); `pipe.register()` registers one now.
204    """
205
206    def __init__(
207        self,
208        data: Dict[str, Any],
209        pipeline: Optional['Pipeline'] = None,
210        options: Optional[Dict[str, Any]] = None,
211        unregistered: Optional[List[mrsm.Pipe]] = None,
212    ):
213        self.raw = data
214        plan = data['plan']
215        self.text: str = data.get('text', '')
216        self.nodes: List[PlannedNode] = [PlannedNode(node) for node in plan['nodes']]
217        self.edges: List[Tuple[str, str]] = [tuple(edge) for edge in plan.get('edges', [])]
218        self.flat: bool = bool(plan.get('flat', False))
219        self.warnings: List[str] = list(data.get('warnings', None) or [])
220        self.unregistered: List[mrsm.Pipe] = list(unregistered or [])
221        self._pipeline = pipeline
222        self._options = options or {}
223        self._unregistered_warnings: List[str] = [
224            _unregistered_warning(pipe, pipeline)
225            for pipe in self.unregistered
226        ]
227        self.warnings.extend(self._unregistered_warnings)
228
229    @property
230    def pipes(self) -> List[mrsm.Pipe]:
231        """The pipes of the plan, in dependency order (`unregistered` pipes are not planned)."""
232        return [node.pipe for node in self.nodes]
233
234    def node(self, pipe: Union[mrsm.Pipe, str]) -> Optional[PlannedNode]:
235        """The planned node of a pipe (or `pipe_ref`)."""
236        pipe_ref = pipe if isinstance(pipe, str) else _pipe_ref(pipe)
237        for node in self.nodes:
238            if node.pipe_ref == pipe_ref:
239                return node
240        return None
241
242    def to_dicts(self) -> List[Dict[str, Any]]:
243        """The plan table as a list of rows."""
244        return [node.to_dict() for node in self.nodes]
245
246    def to_dataframe(
247        self,
248        as_polars: Optional[bool] = None,
249        as_pandas: Optional[bool] = None,
250    ) -> Any:
251        """
252        The plan table (`node`, `action`, `bounds`, `chunks`, `reason`) as a DataFrame of the
253        `meerschaum:dataframe` library (Polars for a new root); `as_polars=True` or
254        `as_pandas=True` asks for one.
255        """
256        from meerschaum.utils.dataframe import resolve_dataframe_library, ensure_dataframe_library
257        columns = ['node', 'action', 'bounds', 'chunks', 'reason']
258        library = resolve_dataframe_library(as_polars=as_polars, as_pandas=as_pandas)
259        module = ensure_dataframe_library(library)
260        if library == 'polars':
261            schema = {
262                'node': module.String,
263                'action': module.String,
264                'bounds': module.String,
265                'chunks': module.Int64,
266                'reason': module.String,
267            }
268            rows = [{col: row.get(col) for col in columns} for row in self.to_dicts()]
269            return module.DataFrame(rows, schema=schema, strict=False)
270        return module.DataFrame(self.to_dicts(), columns=columns)
271
272    def execute(self, **kwargs: Any) -> 'RunResult':
273        """
274        Run the pipeline this plan came from with the options it was planned with
275        (the engine re-plans against current state as the run proceeds).
276        """
277        if self._pipeline is None:
278            raise ValueError("This plan does not belong to a pipeline.")
279        options = {**self._options, **kwargs}
280        return self._pipeline.sync(**options)
281
282    def __len__(self) -> int:
283        return len(self.nodes)
284
285    def __iter__(self):
286        return iter(self.nodes)
287
288    def __str__(self) -> str:
289        if not self._unregistered_warnings:
290            return self.text
291        text = self.text if (not self.text or self.text.endswith('\n')) else self.text + '\n'
292        return text + ''.join(f"warning: {warning}\n" for warning in self._unregistered_warnings)
293
294    def __repr__(self) -> str:
295        return f"Plan({len(self.nodes)} nodes)"

A dry run: what the engine would do, with no data movement.

str(plan) is the plan table (node | action | bounds | chunks | reason, * marks a provisional plan).

Attributes
  • nodes (List[PlannedNode]): The nodes in dependency order.
  • edges (List[Tuple[str, str]]): The (parent, child) pairs inside the selection.
  • flat (bool): True for a selection with no traversal and no dependencies between its members: every node syncs on its own, like mrsm sync pipes.
  • warnings (List[str]): Graph and selection warnings (including one per member pipe that is not registered).
  • unregistered (List[mrsm.Pipe]): Member pipes that are not registered on their instance, so the engine cannot plan them yet. Pipeline.sync() registers them first (except for a saved Pipeline, whose members must already be registered); pipe.register() registers one now.
Plan( data: Dict[str, Any], pipeline: Pipeline | None = None, options: Dict[str, Any] | None = None, unregistered: List[Pipe] | None = None) 
206    def __init__(
207        self,
208        data: Dict[str, Any],
209        pipeline: Optional['Pipeline'] = None,
210        options: Optional[Dict[str, Any]] = None,
211        unregistered: Optional[List[mrsm.Pipe]] = None,
212    ):
213        self.raw = data
214        plan = data['plan']
215        self.text: str = data.get('text', '')
216        self.nodes: List[PlannedNode] = [PlannedNode(node) for node in plan['nodes']]
217        self.edges: List[Tuple[str, str]] = [tuple(edge) for edge in plan.get('edges', [])]
218        self.flat: bool = bool(plan.get('flat', False))
219        self.warnings: List[str] = list(data.get('warnings', None) or [])
220        self.unregistered: List[mrsm.Pipe] = list(unregistered or [])
221        self._pipeline = pipeline
222        self._options = options or {}
223        self._unregistered_warnings: List[str] = [
224            _unregistered_warning(pipe, pipeline)
225            for pipe in self.unregistered
226        ]
227        self.warnings.extend(self._unregistered_warnings)
raw
text: str
nodes: List[meerschaum.core.Pipeline.PlannedNode]
edges: List[Tuple[str, str]]
flat: bool
warnings: List[str]
unregistered: List[Pipe]
pipes: List[Pipe] 
229    @property
230    def pipes(self) -> List[mrsm.Pipe]:
231        """The pipes of the plan, in dependency order (`unregistered` pipes are not planned)."""
232        return [node.pipe for node in self.nodes]

The pipes of the plan, in dependency order (unregistered pipes are not planned).

def node( self, pipe: Pipe | str) -> meerschaum.core.Pipeline.PlannedNode | None: 
234    def node(self, pipe: Union[mrsm.Pipe, str]) -> Optional[PlannedNode]:
235        """The planned node of a pipe (or `pipe_ref`)."""
236        pipe_ref = pipe if isinstance(pipe, str) else _pipe_ref(pipe)
237        for node in self.nodes:
238            if node.pipe_ref == pipe_ref:
239                return node
240        return None

The planned node of a pipe (or pipe_ref).

def to_dicts(self) -> List[Dict[str, Any]]: 
242    def to_dicts(self) -> List[Dict[str, Any]]:
243        """The plan table as a list of rows."""
244        return [node.to_dict() for node in self.nodes]

The plan table as a list of rows.

def to_dataframe( self, as_polars: bool | None = None, as_pandas: bool | None = None) -> Any: 
246    def to_dataframe(
247        self,
248        as_polars: Optional[bool] = None,
249        as_pandas: Optional[bool] = None,
250    ) -> Any:
251        """
252        The plan table (`node`, `action`, `bounds`, `chunks`, `reason`) as a DataFrame of the
253        `meerschaum:dataframe` library (Polars for a new root); `as_polars=True` or
254        `as_pandas=True` asks for one.
255        """
256        from meerschaum.utils.dataframe import resolve_dataframe_library, ensure_dataframe_library
257        columns = ['node', 'action', 'bounds', 'chunks', 'reason']
258        library = resolve_dataframe_library(as_polars=as_polars, as_pandas=as_pandas)
259        module = ensure_dataframe_library(library)
260        if library == 'polars':
261            schema = {
262                'node': module.String,
263                'action': module.String,
264                'bounds': module.String,
265                'chunks': module.Int64,
266                'reason': module.String,
267            }
268            rows = [{col: row.get(col) for col in columns} for row in self.to_dicts()]
269            return module.DataFrame(rows, schema=schema, strict=False)
270        return module.DataFrame(self.to_dicts(), columns=columns)

The plan table (node, action, bounds, chunks, reason) as a DataFrame of the meerschaum:dataframe library (Polars for a new root); as_polars=True or as_pandas=True asks for one.

def execute(self, **kwargs: Any) -> RunResult: 
272    def execute(self, **kwargs: Any) -> 'RunResult':
273        """
274        Run the pipeline this plan came from with the options it was planned with
275        (the engine re-plans against current state as the run proceeds).
276        """
277        if self._pipeline is None:
278            raise ValueError("This plan does not belong to a pipeline.")
279        options = {**self._options, **kwargs}
280        return self._pipeline.sync(**options)

Run the pipeline this plan came from with the options it was planned with (the engine re-plans against current state as the run proceeds).

class RunResult(builtins.tuple): 
321class RunResult(tuple):
322    """
323    The result of an engine run: a `SuccessTuple` (`success, message = run` still unpacks)
324    with the run's details. `pipe.sync(parents=..., children=...)`, `Pipeline.sync()` and
325    `Plan.execute()` return one.
326
327    Attributes
328    ----------
329    success: bool
330        Whether the run succeeded (a failed node fails the run).
331
332    message: str
333        The run summary.
334
335    run_id: Optional[str]
336        The ID in the run history (`Pipeline.get_run(run_id)`); `None` if the run never started.
337
338    plan: Optional[Plan]
339        The plan that was executed.
340
341    inserted, updated, upserted, deleted: Optional[int]
342        Rows inserted (updated, ...) by every node of the run together, like the same fields of
343        the `SyncResult` a plain `pipe.sync()` returns. `None` if the run never started.
344
345    duration: float
346        How long the run took, in wall-clock seconds (nodes that run in parallel overlap, so
347        this is not the sum of the nodes' durations).
348
349    outcomes: List[Dict[str, Any]]
350        One entry per node that synced, in completion order: `pipe_ref`, `pipe` (`str(pipe)`)
351        and `result` (`success`, `message`, `inserted`, `updated`, `upserted`, `deleted`,
352        `changed_intervals`, `duration` (float seconds, or `None`), `begin`, `end`).
353
354    critical_path: List[str]
355        The `pipe_ref`s of the plan's critical path.
356
357    warnings: List[str]
358        Graph, selection and state warnings.
359
360    record: Optional[Dict[str, Any]]
361        The persisted run row and node rows.
362
363    callback_error: Optional[Exception]
364        The exception an `on_event` callback raised (the run continued without it), else
365        `None`.
366    """
367
368    success: bool
369    message: str
370
371    def __new__(cls, success: bool, message: str, data: Optional[Dict[str, Any]] = None, **kwargs):
372        self = super().__new__(cls, (bool(success), message))
373        data = data or {}
374        self.success = bool(success)
375        self.message = message
376        self.run_id = data.get('run_id', None)
377        self.outcomes = [with_seconds(outcome) for outcome in data.get('outcomes', None) or []]
378        self.critical_path = list(data.get('critical_path', None) or [])
379        self.warnings = list(data.get('warnings', None) or [])
380        self.duration = float(data.get('duration_seconds', 0.0) or 0.0)
381        self.record = data.get('record', None)
382        self.plan = Plan(data, **kwargs) if data.get('plan', None) else None
383        ran = 'outcomes' in data
384        for field in _RUN_TOTALS:
385            setattr(self, field, _total(self.outcomes, field) if ran else None)
386        self._data = data
387        self._instance_request: Optional[Dict[str, Any]] = None
388        self.callback_error: Optional[BaseException] = None
389        return self
390
391    def __reduce__(self):
392        ### The details are JSON data; the pipeline back-reference is dropped.
393        return (_rebuild_run_result, (self.success, self.message, self._data))
394
395    @property
396    def result(self) -> 'RunResult':
397        """The run result (this object: it is already `SuccessTuple`-compatible)."""
398        return self
399
400    def get_outcome(self, pipe: Union[mrsm.Pipe, str]) -> Optional[Dict[str, Any]]:
401        """The outcome of a node of the run."""
402        pipe_ref = pipe if isinstance(pipe, str) else _pipe_ref(pipe)
403        for outcome in self.outcomes:
404            if outcome['pipe_ref'] == pipe_ref:
405                return outcome
406        return None
407
408    def to_dict(self) -> Dict[str, Any]:
409        """The run's summary fields as a dictionary."""
410        return {
411            'success': self.success,
412            'message': self.message,
413            'run_id': self.run_id,
414            **{field: getattr(self, field) for field in _RUN_TOTALS},
415            'duration': self.duration,
416        }
417
418    def events(self, after_seq: Optional[int] = None) -> List[Dict[str, Any]]:
419        """
420        The run's persisted events (`run_started`, `node_started`, `node_finished`,
421        `run_finished`, ...) in order, read from the instance after the run
422        (`system.engine.events` sets how fine-grained they are). Each row has the keys of a live
423        `on_event` event (`event`, `run_id`, `seq`, `ts`, `pipe_ref` and the event's own
424        fields) plus `payload`, the event's own fields as one dict. `ts` is the ISO 8601 text
425        the instance stored.
426
427        To receive every event live while the run is going, pass `on_event=` to
428        `Pipeline.sync()` (or `pipe.sync(parents=..., children=...)`).
429        """
430        if self.run_id is None or self._instance_request is None:
431            return []
432        request = {**self._instance_request, 'run_id': self.run_id}
433        if after_seq is not None:
434            request['after_seq'] = after_seq
435        return _decode_events(call('engine_run_events', request))
436
437    def __repr__(self) -> str:
438        return f"RunResult(success={self.success}, run_id={self.run_id!r}, message={self.message!r})"

The result of an engine run: a SuccessTuple (success, message = run still unpacks) with the run's details. pipe.sync(parents=..., children=...), Pipeline.sync() and Plan.execute() return one.

Attributes
  • success (bool): Whether the run succeeded (a failed node fails the run).
  • message (str): The run summary.
  • run_id (Optional[str]): The ID in the run history (Pipeline.get_run(run_id)); None if the run never started.
  • plan (Optional[Plan]): The plan that was executed.
  • inserted, updated, upserted, deleted (Optional[int]): Rows inserted (updated, ...) by every node of the run together, like the same fields of the SyncResult a plain pipe.sync() returns. None if the run never started.
  • duration (float): How long the run took, in wall-clock seconds (nodes that run in parallel overlap, so this is not the sum of the nodes' durations).
  • outcomes (List[Dict[str, Any]]): One entry per node that synced, in completion order: pipe_ref, pipe (str(pipe)) and result (success, message, inserted, updated, upserted, deleted, changed_intervals, duration (float seconds, or None), begin, end).
  • critical_path (List[str]): The pipe_refs of the plan's critical path.
  • warnings (List[str]): Graph, selection and state warnings.
  • record (Optional[Dict[str, Any]]): The persisted run row and node rows.
  • callback_error (Optional[Exception]): The exception an on_event callback raised (the run continued without it), else None.
success: bool
message: str
result: RunResult 
395    @property
396    def result(self) -> 'RunResult':
397        """The run result (this object: it is already `SuccessTuple`-compatible)."""
398        return self

The run result (this object: it is already SuccessTuple-compatible).

def get_outcome(self, pipe: Pipe | str) -> Dict[str, Any] | None: 
400    def get_outcome(self, pipe: Union[mrsm.Pipe, str]) -> Optional[Dict[str, Any]]:
401        """The outcome of a node of the run."""
402        pipe_ref = pipe if isinstance(pipe, str) else _pipe_ref(pipe)
403        for outcome in self.outcomes:
404            if outcome['pipe_ref'] == pipe_ref:
405                return outcome
406        return None

The outcome of a node of the run.

def to_dict(self) -> Dict[str, Any]: 
408    def to_dict(self) -> Dict[str, Any]:
409        """The run's summary fields as a dictionary."""
410        return {
411            'success': self.success,
412            'message': self.message,
413            'run_id': self.run_id,
414            **{field: getattr(self, field) for field in _RUN_TOTALS},
415            'duration': self.duration,
416        }

The run's summary fields as a dictionary.

def events(self, after_seq: int | None = None) -> List[Dict[str, Any]]: 
418    def events(self, after_seq: Optional[int] = None) -> List[Dict[str, Any]]:
419        """
420        The run's persisted events (`run_started`, `node_started`, `node_finished`,
421        `run_finished`, ...) in order, read from the instance after the run
422        (`system.engine.events` sets how fine-grained they are). Each row has the keys of a live
423        `on_event` event (`event`, `run_id`, `seq`, `ts`, `pipe_ref` and the event's own
424        fields) plus `payload`, the event's own fields as one dict. `ts` is the ISO 8601 text
425        the instance stored.
426
427        To receive every event live while the run is going, pass `on_event=` to
428        `Pipeline.sync()` (or `pipe.sync(parents=..., children=...)`).
429        """
430        if self.run_id is None or self._instance_request is None:
431            return []
432        request = {**self._instance_request, 'run_id': self.run_id}
433        if after_seq is not None:
434            request['after_seq'] = after_seq
435        return _decode_events(call('engine_run_events', request))

The run's persisted events (run_started, node_started, node_finished, run_finished, ...) in order, read from the instance after the run (system.engine.events sets how fine-grained they are). Each row has the keys of a live on_event event (event, run_id, seq, ts, pipe_ref and the event's own fields) plus payload, the event's own fields as one dict. ts is the ISO 8601 text the instance stored.

To receive every event live while the run is going, pass on_event= to Pipeline.sync() (or pipe.sync(parents=..., children=...)).

class SyncResult(builtins.tuple): 
 39class SyncResult(tuple):
 40    """
 41    The result of `Pipe.sync()`: a `SuccessTuple` (`(success, message)`, so
 42    `success, msg = pipe.sync()` unpacks as ever) with the structured fields of the sync.
 43
 44    The fields are `None` when the sync did not report them (a plugin's own `sync()`,
 45    the Python implementation, a failure before the data was written).
 46
 47    Attributes
 48    ----------
 49    success: bool
 50        Whether the sync succeeded.
 51
 52    message: str
 53        The message.
 54
 55    inserted: Optional[int]
 56        Rows inserted.
 57
 58    updated: Optional[int]
 59        Rows updated.
 60
 61    upserted: Optional[int]
 62        Rows upserted.
 63
 64    deleted: Optional[int]
 65        Rows deleted.
 66
 67    changed_intervals: Optional[List[Tuple[Any, Any]]]
 68        The `(begin, end)` intervals of the datetime axis that changed: the epoch-aligned grid
 69        cells (one chunk interval wide) that hold the rows the sync handed to the instance,
 70        merged into runs. This is a superset of the rows that actually changed (rows that
 71        turned out to be unchanged duplicates are inside it too). An open end is `None`.
 72
 73    duration: Optional[float]
 74        Seconds the sync took.
 75
 76    begin: Union[datetime, int, None]
 77        The lower bound of the sync, if one was given.
 78
 79    end: Union[datetime, int, None]
 80        The upper bound of the sync, if one was given.
 81    """
 82
 83    _FIELDS = (
 84        'inserted', 'updated', 'upserted', 'deleted',
 85        'changed_intervals', 'duration', 'begin', 'end',
 86    )
 87
 88    success: bool
 89    message: str
 90
 91    def __new__(cls, success: bool, message: str = 'Success', **fields: Any):
 92        self = super().__new__(cls, (success, message))
 93        self.success = success
 94        self.message = message
 95        for name in cls._FIELDS:
 96            setattr(self, name, fields.pop(name, None))
 97        if fields:
 98            raise TypeError(f"Unexpected sync result fields: {', '.join(sorted(fields))}.")
 99        return self
100
101    @classmethod
102    def from_tuple(cls, result: Any) -> Any:
103        """
104        Return `result` as a `SyncResult` if it is a plain `(bool, str)` tuple,
105        otherwise return it as it is (e.g. a `RunResult` or a plugin's own tuple).
106        """
107        if type(result) is not tuple or len(result) != 2:
108            return result
109        return cls(result[0], result[1])
110
111    def to_dict(self) -> Dict[str, Any]:
112        """
113        Return the result's fields as a dictionary.
114        """
115        return {
116            'success': self.success,
117            'message': self.message,
118            **{name: getattr(self, name) for name in self._FIELDS},
119        }
120
121    def __reduce__(self):
122        return _rebuild_sync_result, (self.to_dict(),)

The result of Pipe.sync(): a SuccessTuple ((success, message), so success, msg = pipe.sync() unpacks as ever) with the structured fields of the sync.

The fields are None when the sync did not report them (a plugin's own sync(), the Python implementation, a failure before the data was written).

Attributes
  • success (bool): Whether the sync succeeded.
  • message (str): The message.
  • inserted (Optional[int]): Rows inserted.
  • updated (Optional[int]): Rows updated.
  • upserted (Optional[int]): Rows upserted.
  • deleted (Optional[int]): Rows deleted.
  • changed_intervals (Optional[List[Tuple[Any, Any]]]): The (begin, end) intervals of the datetime axis that changed: the epoch-aligned grid cells (one chunk interval wide) that hold the rows the sync handed to the instance, merged into runs. This is a superset of the rows that actually changed (rows that turned out to be unchanged duplicates are inside it too). An open end is None.
  • duration (Optional[float]): Seconds the sync took.
  • begin (Union[datetime, int, None]): The lower bound of the sync, if one was given.
  • end (Union[datetime, int, None]): The upper bound of the sync, if one was given.
success: bool
message: str
@classmethod
def from_tuple(cls, result: Any) -> Any: 
101    @classmethod
102    def from_tuple(cls, result: Any) -> Any:
103        """
104        Return `result` as a `SyncResult` if it is a plain `(bool, str)` tuple,
105        otherwise return it as it is (e.g. a `RunResult` or a plugin's own tuple).
106        """
107        if type(result) is not tuple or len(result) != 2:
108            return result
109        return cls(result[0], result[1])

Return result as a SyncResult if it is a plain (bool, str) tuple, otherwise return it as it is (e.g. a RunResult or a plugin's own tuple).

def to_dict(self) -> Dict[str, Any]: 
111    def to_dict(self) -> Dict[str, Any]:
112        """
113        Return the result's fields as a dictionary.
114        """
115        return {
116            'success': self.success,
117            'message': self.message,
118            **{name: getattr(self, name) for name in self._FIELDS},
119        }

Return the result's fields as a dictionary.

class Team: 
 44class Team:
 45    """
 46    A team of users on an instance (the API server's user store).
 47
 48    Every user is a member of the built-in team `Everyone`, which cannot be deleted or left.
 49    Administrators create more teams; a user can be in several, as a `'member'` or a
 50    `'manager'`. Managers add and remove members of their team and manage the access of its
 51    pipes. Renaming a team renames it in the pipes and jobs that name it; deleting one moves
 52    them to `Everyone`.
 53
 54    Parameters
 55    ----------
 56    name: str
 57        The team's name (the username rules; names are unique without case).
 58
 59    instance: Union[str, InstanceConnector, None], default None
 60        The instance that holds the team (the configured instance by default). Through an
 61        `api:` instance the server checks who may change the team.
 62
 63    members: Optional[Iterable[str]], default None
 64        Members to add (in memory, until `register()` or `edit()`).
 65
 66    managers: Optional[Iterable[str]], default None
 67        Managers to add (in memory, until `register()` or `edit()`).
 68
 69    attributes: Optional[Dict[str, Any]], default None
 70        Free-form attributes.
 71
 72    Examples
 73    --------
 74    >>> team = mrsm.Team('ops', instance='sql:local', members=['alice'], managers=['mia'])
 75    >>> team.register()
 76    (True, "Registered team 'ops'.")
 77    >>> team.add_members('bob')
 78    (True, "Edited team 'ops'.")
 79    >>> team.members
 80    ['alice', 'mia', 'bob']
 81    >>> team.managers
 82    ['mia']
 83    """
 84
 85    def __init__(
 86        self,
 87        name: str,
 88        instance: Union[str, 'mrsm.connectors.InstanceConnector', None] = None,
 89        members: Optional[Iterable[Any]] = None,
 90        managers: Optional[Iterable[Any]] = None,
 91        attributes: Optional[Dict[str, Any]] = None,
 92        team_id: Optional[str] = None,
 93    ):
 94        self.name = str(name).strip()
 95        if instance is None:
 96            from meerschaum.config import get_config
 97            instance = get_config('meerschaum', 'instance', patch=True)
 98        if not isinstance(instance, str):
 99            self._instance_connector = instance
100            self._instance_keys = str(instance)
101        else:
102            self._instance_keys = instance
103        self._id = team_id
104        self._roles: Dict[str, str] = {}
105        self._attributes: Dict[str, Any] = dict(attributes or {})
106        self._loaded = False
107        for username in _usernames(members or []):
108            self._roles[username] = 'member'
109        for username in _usernames(managers or []):
110            self._roles[username] = 'manager'
111        self._pending = bool(self._roles) or bool(self._attributes)
112
113    # --------------------------------------------------------------------------------------
114    # The instance
115    # --------------------------------------------------------------------------------------
116
117    @property
118    def instance_keys(self) -> str:
119        """The keys of the instance holding the team."""
120        return self._instance_keys
121
122    @property
123    def instance_connector(self) -> 'mrsm.connectors.InstanceConnector':
124        """The instance connector holding the team."""
125        if '_instance_connector' not in self.__dict__:
126            from meerschaum.connectors.parse import parse_instance_keys
127            self._instance_connector = parse_instance_keys(self._instance_keys)
128        return self._instance_connector
129
130    @property
131    def is_everyone(self) -> bool:
132        """Whether this is the built-in team `Everyone`."""
133        return self.name.lower() == EVERYONE.lower() or not self.name
134
135    def _load(self, refresh: bool = False, debug: bool = False) -> Optional[Dict[str, Any]]:
136        """Read the team from the instance (its record, or `None` when it does not exist)."""
137        if self._loaded and not refresh:
138            return None
139        record = self.instance_connector.get_team(self.name, debug=debug)
140        self._loaded = True
141        if not record:
142            return None
143        if not self._pending:
144            self._apply_record(record)
145        else:
146            ### Names compare without case: the stored spelling wins.
147            self.name = str(record.get('name', None) or self.name)
148            self._id = record.get('id', None) or self._id
149        return record
150
151    def _apply_record(self, record: Dict[str, Any]) -> None:
152        self.name = str(record.get('name', self.name) or self.name)
153        self._id = record.get('id', None) or record.get('team_id', None) or self._id
154        members = record.get('members', None) or {}
155        self._roles = {
156            str(username): _role(role)
157            for username, role in (
158                members.items() if isinstance(members, dict) else ((m, 'member') for m in members)
159            )
160        }
161        self._attributes = dict(record.get('attributes', None) or {})
162
163    def refresh(self, debug: bool = False) -> 'Team':
164        """Read the team again from the instance (in-memory changes are dropped)."""
165        self._pending = False
166        self._loaded = False
167        self._load(refresh=True, debug=debug)
168        return self
169
170    def exists(self, debug: bool = False) -> bool:
171        """
172        Whether the team exists on the instance (`Everyone` always does). An existing team's
173        `name` takes its stored spelling (`Team('OPS').name` is `'ops'`).
174        """
175        if self.is_everyone:
176            return True
177        return self._load(refresh=True, debug=debug) is not None
178
179    # --------------------------------------------------------------------------------------
180    # Members
181    # --------------------------------------------------------------------------------------
182
183    @property
184    def id(self) -> Optional[str]:
185        """The team's id (`None` before it is registered)."""
186        self._load()
187        return self._id
188
189    @property
190    def roles(self) -> Dict[str, str]:
191        """`{username: 'member' | 'manager'}` of the listed members."""
192        self._load()
193        return dict(self._roles)
194
195    @property
196    def members(self) -> List[str]:
197        """
198        The usernames of the members (managers included). For `Everyone`, every registered
199        user.
200        """
201        self._load()
202        if self.is_everyone:
203            users = list(self.instance_connector.get_users() or [])
204            return users + [u for u in self._roles if u not in users]
205        return list(self._roles)
206
207    @property
208    def managers(self) -> List[str]:
209        """The usernames of the managers."""
210        self._load()
211        return [username for username, role in self._roles.items() if role == 'manager']
212
213    @property
214    def attributes(self) -> Dict[str, Any]:
215        """The team's free-form attributes (edit them, then call `edit()`)."""
216        self._load()
217        self._pending = True
218        return self._attributes
219
220    def role_of(self, user: Any) -> Optional[str]:
221        """A user's role in the team (`'member'` for anyone in `Everyone`), or `None`."""
222        username = str(getattr(user, 'username', user))
223        role = self.roles.get(username, None)
224        if role is None and self.is_everyone:
225            return 'member'
226        return role
227
228    def to_dict(self) -> Dict[str, Any]:
229        """The team as stored: `{'id', 'name', 'members': {username: role}, 'attributes'}`."""
230        return {
231            'id': self._id,
232            'name': EVERYONE if self.is_everyone else self.name,
233            'members': dict(self._roles),
234            'attributes': dict(self._attributes),
235        }
236
237    def _write(self, debug: bool = False) -> mrsm.SuccessTuple:
238        record = self.to_dict()
239        result = self.instance_connector.edit_team(record, debug=debug)
240        if result[0]:
241            self._pending = False
242        return result
243
244    def add_members(self, *usernames: Any, role: str = 'member', debug: bool = False) -> mrsm.SuccessTuple:
245        """
246        Add users to the team as `role` (`'member'` or `'manager'`; an existing member takes the
247        new role). A registered team is saved at once.
248        """
249        names = _usernames(usernames)
250        if not names:
251            return False, "No users to add."
252        role = _role(role)
253        self._load(debug=debug)
254        for username in names:
255            self._roles[username] = role
256        self._pending = True
257        if not self.is_everyone and not self._id:
258            return True, f"Added {len(names)} user(s) to team '{self.name}' (not registered yet)."
259        return self._write(debug=debug)
260
261    def add_managers(self, *usernames: Any, debug: bool = False) -> mrsm.SuccessTuple:
262        """Make users managers of the team (`add_members(..., role='manager')`)."""
263        return self.add_members(*usernames, role='manager', debug=debug)
264
265    def remove_members(self, *usernames: Any, debug: bool = False) -> mrsm.SuccessTuple:
266        """
267        Remove users from the team (a manager of `Everyone` becomes a plain member: nobody leaves
268        `Everyone`). A registered team is saved at once.
269        """
270        names = _usernames(usernames)
271        if not names:
272            return False, "No users to remove."
273        self._load(debug=debug)
274        missing = [u for u in names if u not in self._roles]
275        if missing and not self.is_everyone:
276            return False, (
277                f"Not in team '{self.name}': " + ', '.join(f"'{u}'" for u in missing) + "."
278            )
279        for username in names:
280            self._roles.pop(username, None)
281        self._pending = True
282        if not self.is_everyone and not self._id:
283            return True, f"Removed {len(names)} user(s) from team '{self.name}' (not registered yet)."
284        return self._write(debug=debug)
285
286    # --------------------------------------------------------------------------------------
287    # Lifecycle
288    # --------------------------------------------------------------------------------------
289
290    def register(self, debug: bool = False) -> mrsm.SuccessTuple:
291        """Register the team (administrators only through the API)."""
292        if self.is_everyone:
293            return False, f"'{EVERYONE}' is the built-in team of every user: edit it instead."
294        record = self.to_dict()
295        result = self.instance_connector.register_team(record, debug=debug)
296        if result[0]:
297            self._id = record.get('id', None) or self._id
298            self._pending = False
299            self._loaded = True
300        return result
301
302    def edit(self, debug: bool = False) -> mrsm.SuccessTuple:
303        """Save the team's members and attributes."""
304        self._load(debug=debug)
305        return self._write(debug=debug)
306
307    def rename(self, new_name: str, debug: bool = False) -> mrsm.SuccessTuple:
308        """Rename the team (the pipes and jobs that name it follow)."""
309        old_name = self.name
310        result = self.instance_connector.rename_team(old_name, str(new_name).strip(), debug=debug)
311        if result[0]:
312            self.name = str(new_name).strip()
313            _rename_jobs(old_name, self.name, debug=debug)
314        return result
315
316    def delete(self, debug: bool = False) -> mrsm.SuccessTuple:
317        """Delete the team (its pipes and jobs move to `Everyone`)."""
318        result = self.instance_connector.delete_team(self.name, debug=debug)
319        if result[0]:
320            _rename_jobs(self.name, None, debug=debug)
321            self._id = None
322            self._loaded = False
323        return result
324
325    def get_pipes(self, debug: bool = False) -> List['mrsm.Pipe']:
326        """The pipes (and saved Pipelines' registry pipes) whose access names this team."""
327        from meerschaum.core.access import get_pipe_access, same_team
328        pipes = mrsm.get_pipes(instance=self.instance_connector, as_list=True, debug=debug)
329        return [
330            pipe for pipe in pipes
331            if same_team(get_pipe_access(pipe).get('team', None), None if self.is_everyone else self.name)
332        ]
333
334    def __repr__(self) -> str:
335        return f"Team('{self.name}')"
336
337    def __str__(self) -> str:
338        return self.name
339
340    def __eq__(self, other: Any) -> bool:
341        from meerschaum.core.access import same_team
342        return (
343            isinstance(other, Team)
344            and same_team(self.name, other.name)
345            and self.instance_keys == other.instance_keys
346        )
347
348    def __hash__(self) -> int:
349        return hash((self.name.lower(), self.instance_keys))

A team of users on an instance (the API server's user store).

Every user is a member of the built-in team Everyone, which cannot be deleted or left. Administrators create more teams; a user can be in several, as a 'member' or a 'manager'. Managers add and remove members of their team and manage the access of its pipes. Renaming a team renames it in the pipes and jobs that name it; deleting one moves them to Everyone.

Parameters
  • name (str): The team's name (the username rules; names are unique without case).
  • instance (Union[str, InstanceConnector, None], default None): The instance that holds the team (the configured instance by default). Through an api: instance the server checks who may change the team.
  • members (Optional[Iterable[str]], default None): Members to add (in memory, until register() or edit()).
  • managers (Optional[Iterable[str]], default None): Managers to add (in memory, until register() or edit()).
  • attributes (Optional[Dict[str, Any]], default None): Free-form attributes.
Examples
>>> team = mrsm.Team('ops', instance='sql:local', members=['alice'], managers=['mia'])
>>> team.register()
(True, "Registered team 'ops'.")
>>> team.add_members('bob')
(True, "Edited team 'ops'.")
>>> team.members
['alice', 'mia', 'bob']
>>> team.managers
['mia']
Team( name: str, instance: str | InstanceConnector | None = None, members: Iterable[Any] | None = None, managers: Iterable[Any] | None = None, attributes: Dict[str, Any] | None = None, team_id: str | None = None) 
 85    def __init__(
 86        self,
 87        name: str,
 88        instance: Union[str, 'mrsm.connectors.InstanceConnector', None] = None,
 89        members: Optional[Iterable[Any]] = None,
 90        managers: Optional[Iterable[Any]] = None,
 91        attributes: Optional[Dict[str, Any]] = None,
 92        team_id: Optional[str] = None,
 93    ):
 94        self.name = str(name).strip()
 95        if instance is None:
 96            from meerschaum.config import get_config
 97            instance = get_config('meerschaum', 'instance', patch=True)
 98        if not isinstance(instance, str):
 99            self._instance_connector = instance
100            self._instance_keys = str(instance)
101        else:
102            self._instance_keys = instance
103        self._id = team_id
104        self._roles: Dict[str, str] = {}
105        self._attributes: Dict[str, Any] = dict(attributes or {})
106        self._loaded = False
107        for username in _usernames(members or []):
108            self._roles[username] = 'member'
109        for username in _usernames(managers or []):
110            self._roles[username] = 'manager'
111        self._pending = bool(self._roles) or bool(self._attributes)
name
instance_keys: str 
117    @property
118    def instance_keys(self) -> str:
119        """The keys of the instance holding the team."""
120        return self._instance_keys

The keys of the instance holding the team.

instance_connector: InstanceConnector 
122    @property
123    def instance_connector(self) -> 'mrsm.connectors.InstanceConnector':
124        """The instance connector holding the team."""
125        if '_instance_connector' not in self.__dict__:
126            from meerschaum.connectors.parse import parse_instance_keys
127            self._instance_connector = parse_instance_keys(self._instance_keys)
128        return self._instance_connector

The instance connector holding the team.

is_everyone: bool 
130    @property
131    def is_everyone(self) -> bool:
132        """Whether this is the built-in team `Everyone`."""
133        return self.name.lower() == EVERYONE.lower() or not self.name

Whether this is the built-in team Everyone.

def refresh(self, debug: bool = False) -> Team: 
163    def refresh(self, debug: bool = False) -> 'Team':
164        """Read the team again from the instance (in-memory changes are dropped)."""
165        self._pending = False
166        self._loaded = False
167        self._load(refresh=True, debug=debug)
168        return self

Read the team again from the instance (in-memory changes are dropped).

def exists(self, debug: bool = False) -> bool: 
170    def exists(self, debug: bool = False) -> bool:
171        """
172        Whether the team exists on the instance (`Everyone` always does). An existing team's
173        `name` takes its stored spelling (`Team('OPS').name` is `'ops'`).
174        """
175        if self.is_everyone:
176            return True
177        return self._load(refresh=True, debug=debug) is not None

Whether the team exists on the instance (Everyone always does). An existing team's name takes its stored spelling (Team('OPS').name is 'ops').

id: str | None 
183    @property
184    def id(self) -> Optional[str]:
185        """The team's id (`None` before it is registered)."""
186        self._load()
187        return self._id

The team's id (None before it is registered).

roles: Dict[str, str] 
189    @property
190    def roles(self) -> Dict[str, str]:
191        """`{username: 'member' | 'manager'}` of the listed members."""
192        self._load()
193        return dict(self._roles)

{username: 'member' | 'manager'} of the listed members.

members: List[str] 
195    @property
196    def members(self) -> List[str]:
197        """
198        The usernames of the members (managers included). For `Everyone`, every registered
199        user.
200        """
201        self._load()
202        if self.is_everyone:
203            users = list(self.instance_connector.get_users() or [])
204            return users + [u for u in self._roles if u not in users]
205        return list(self._roles)

The usernames of the members (managers included). For Everyone, every registered user.

managers: List[str] 
207    @property
208    def managers(self) -> List[str]:
209        """The usernames of the managers."""
210        self._load()
211        return [username for username, role in self._roles.items() if role == 'manager']

The usernames of the managers.

attributes: Dict[str, Any] 
213    @property
214    def attributes(self) -> Dict[str, Any]:
215        """The team's free-form attributes (edit them, then call `edit()`)."""
216        self._load()
217        self._pending = True
218        return self._attributes

The team's free-form attributes (edit them, then call edit()).

def role_of(self, user: Any) -> str | None: 
220    def role_of(self, user: Any) -> Optional[str]:
221        """A user's role in the team (`'member'` for anyone in `Everyone`), or `None`."""
222        username = str(getattr(user, 'username', user))
223        role = self.roles.get(username, None)
224        if role is None and self.is_everyone:
225            return 'member'
226        return role

A user's role in the team ('member' for anyone in Everyone), or None.

def to_dict(self) -> Dict[str, Any]: 
228    def to_dict(self) -> Dict[str, Any]:
229        """The team as stored: `{'id', 'name', 'members': {username: role}, 'attributes'}`."""
230        return {
231            'id': self._id,
232            'name': EVERYONE if self.is_everyone else self.name,
233            'members': dict(self._roles),
234            'attributes': dict(self._attributes),
235        }

The team as stored: {'id', 'name', 'members': {username: role}, 'attributes'}.

def add_members( self, *usernames: Any, role: str = 'member', debug: bool = False) -> Tuple[bool, str]: 
244    def add_members(self, *usernames: Any, role: str = 'member', debug: bool = False) -> mrsm.SuccessTuple:
245        """
246        Add users to the team as `role` (`'member'` or `'manager'`; an existing member takes the
247        new role). A registered team is saved at once.
248        """
249        names = _usernames(usernames)
250        if not names:
251            return False, "No users to add."
252        role = _role(role)
253        self._load(debug=debug)
254        for username in names:
255            self._roles[username] = role
256        self._pending = True
257        if not self.is_everyone and not self._id:
258            return True, f"Added {len(names)} user(s) to team '{self.name}' (not registered yet)."
259        return self._write(debug=debug)

Add users to the team as role ('member' or 'manager'; an existing member takes the new role). A registered team is saved at once.

def add_managers(self, *usernames: Any, debug: bool = False) -> Tuple[bool, str]: 
261    def add_managers(self, *usernames: Any, debug: bool = False) -> mrsm.SuccessTuple:
262        """Make users managers of the team (`add_members(..., role='manager')`)."""
263        return self.add_members(*usernames, role='manager', debug=debug)

Make users managers of the team (add_members(..., role='manager')).

def remove_members(self, *usernames: Any, debug: bool = False) -> Tuple[bool, str]: 
265    def remove_members(self, *usernames: Any, debug: bool = False) -> mrsm.SuccessTuple:
266        """
267        Remove users from the team (a manager of `Everyone` becomes a plain member: nobody leaves
268        `Everyone`). A registered team is saved at once.
269        """
270        names = _usernames(usernames)
271        if not names:
272            return False, "No users to remove."
273        self._load(debug=debug)
274        missing = [u for u in names if u not in self._roles]
275        if missing and not self.is_everyone:
276            return False, (
277                f"Not in team '{self.name}': " + ', '.join(f"'{u}'" for u in missing) + "."
278            )
279        for username in names:
280            self._roles.pop(username, None)
281        self._pending = True
282        if not self.is_everyone and not self._id:
283            return True, f"Removed {len(names)} user(s) from team '{self.name}' (not registered yet)."
284        return self._write(debug=debug)

Remove users from the team (a manager of Everyone becomes a plain member: nobody leaves Everyone). A registered team is saved at once.

def register(self, debug: bool = False) -> Tuple[bool, str]: 
290    def register(self, debug: bool = False) -> mrsm.SuccessTuple:
291        """Register the team (administrators only through the API)."""
292        if self.is_everyone:
293            return False, f"'{EVERYONE}' is the built-in team of every user: edit it instead."
294        record = self.to_dict()
295        result = self.instance_connector.register_team(record, debug=debug)
296        if result[0]:
297            self._id = record.get('id', None) or self._id
298            self._pending = False
299            self._loaded = True
300        return result

Register the team (administrators only through the API).

def edit(self, debug: bool = False) -> Tuple[bool, str]: 
302    def edit(self, debug: bool = False) -> mrsm.SuccessTuple:
303        """Save the team's members and attributes."""
304        self._load(debug=debug)
305        return self._write(debug=debug)

Save the team's members and attributes.

def rename(self, new_name: str, debug: bool = False) -> Tuple[bool, str]: 
307    def rename(self, new_name: str, debug: bool = False) -> mrsm.SuccessTuple:
308        """Rename the team (the pipes and jobs that name it follow)."""
309        old_name = self.name
310        result = self.instance_connector.rename_team(old_name, str(new_name).strip(), debug=debug)
311        if result[0]:
312            self.name = str(new_name).strip()
313            _rename_jobs(old_name, self.name, debug=debug)
314        return result

Rename the team (the pipes and jobs that name it follow).

def delete(self, debug: bool = False) -> Tuple[bool, str]: 
316    def delete(self, debug: bool = False) -> mrsm.SuccessTuple:
317        """Delete the team (its pipes and jobs move to `Everyone`)."""
318        result = self.instance_connector.delete_team(self.name, debug=debug)
319        if result[0]:
320            _rename_jobs(self.name, None, debug=debug)
321            self._id = None
322            self._loaded = False
323        return result

Delete the team (its pipes and jobs move to Everyone).

def get_pipes(self, debug: bool = False) -> List[Pipe]: 
325    def get_pipes(self, debug: bool = False) -> List['mrsm.Pipe']:
326        """The pipes (and saved Pipelines' registry pipes) whose access names this team."""
327        from meerschaum.core.access import get_pipe_access, same_team
328        pipes = mrsm.get_pipes(instance=self.instance_connector, as_list=True, debug=debug)
329        return [
330            pipe for pipe in pipes
331            if same_team(get_pipe_access(pipe).get('team', None), None if self.is_everyone else self.name)
332        ]

The pipes (and saved Pipelines' registry pipes) whose access names this team.

class Plugin: 
  50class Plugin:
  51    """Handle packaging of Meerschaum plugins."""
  52
  53    def __init__(
  54        self,
  55        name: str,
  56        version: Optional[str] = None,
  57        user_id: Optional[int] = None,
  58        required: Optional[List[str]] = None,
  59        attributes: Optional[Dict[str, Any]] = None,
  60        archive_path: Optional[pathlib.Path] = None,
  61        venv_path: Optional[pathlib.Path] = None,
  62        repo_connector: Optional['mrsm.connectors.api.APIConnector'] = None,
  63        repo: Union['mrsm.connectors.api.APIConnector', str, None] = None,
  64    ):
  65        import meerschaum.config.paths as paths
  66        from meerschaum._internal.static import STATIC_CONFIG
  67        sep = STATIC_CONFIG['plugins']['repo_separator']
  68        _repo = None
  69        if sep in name:
  70            try:
  71                name, _repo = name.split(sep)
  72            except Exception as e:
  73                error(f"Invalid plugin name: '{name}'")
  74        self._repo_in_name = _repo
  75
  76        if attributes is None:
  77            attributes = {}
  78        self.name = name
  79        self.attributes = attributes
  80        self.user_id = user_id
  81        self._version = version
  82        if required:
  83            self._required = required
  84        self.archive_path = (
  85            archive_path if archive_path is not None
  86            else paths.PLUGINS_ARCHIVES_RESOURCES_PATH / f"{self.name}.tar.gz"
  87        )
  88        self.venv_path = (
  89            venv_path if venv_path is not None
  90            else paths.VIRTENV_RESOURCES_PATH / self.name
  91        )
  92        self._repo_connector = repo_connector
  93        self._repo_keys = repo
  94
  95
  96    @property
  97    def repo_connector(self):
  98        """
  99        Return the repository connector for this plugin.
 100        NOTE: This imports the `connectors` module, which imports certain plugin modules.
 101        """
 102        if self._repo_connector is None:
 103            from meerschaum.connectors.parse import parse_repo_keys
 104
 105            repo_keys = self._repo_keys or self._repo_in_name
 106            if self._repo_in_name and self._repo_keys and self._repo_keys != self._repo_in_name:
 107                error(
 108                    f"Received inconsistent repos: '{self._repo_in_name}' and '{self._repo_keys}'."
 109                )
 110            repo_connector = parse_repo_keys(repo_keys)
 111            self._repo_connector = repo_connector
 112        return self._repo_connector
 113
 114
 115    @property
 116    def version(self):
 117        """
 118        Return the plugin's module version is defined (`__version__`) if it's defined.
 119        """
 120        if self._version is None:
 121            try:
 122                self._version = self.module.__version__
 123            except Exception as e:
 124                self._version = None
 125        return self._version
 126
 127
 128    @property
 129    def module(self):
 130        """
 131        Return the Python module of the underlying plugin,
 132        or `None` if the plugin is not installed or its import failed
 133        (check `Plugin.import_error` for the swallowed exception).
 134        """
 135        if '_module' not in self.__dict__ or self.__dict__.get('_module', None) is None:
 136            from meerschaum.plugins import _get_loaded_module, _get_plugins_package_name
 137            loaded_module = _get_loaded_module(f'{_get_plugins_package_name()}.{self.name}')
 138            if loaded_module is not None:
 139                self._module = loaded_module
 140                return self._module
 141
 142            if self.__file__ is None:
 143                return None
 144
 145            from meerschaum.plugins import import_plugins
 146            self._module = import_plugins(str(self), warn=False)
 147
 148        return self._module
 149
 150    @property
 151    def import_error(self) -> Union[Exception, None]:
 152        """
 153        Return the exception raised by this plugin's most recent failed import
 154        (i.e. when `Plugin.module` is `None` even though the plugin is installed),
 155        otherwise `None`.
 156        """
 157        from meerschaum.plugins import _plugins_import_errors
 158        return _plugins_import_errors.get(self.name, None)
 159
 160
 161    @property
 162    def __file__(self) -> Union[str, None]:
 163        """
 164        Return the file path (str) of the plugin if it exists, otherwise `None`.
 165        """
 166        if self.__dict__.get('_module', None) is not None:
 167            return self.module.__file__
 168
 169        import meerschaum.config.paths as paths
 170
 171        potential_dir = paths.PLUGINS_RESOURCES_PATH / self.name
 172        if (
 173            potential_dir.exists()
 174            and potential_dir.is_dir()
 175            and (potential_dir / '__init__.py').exists()
 176        ):
 177            return str((potential_dir / '__init__.py').as_posix())
 178
 179        potential_file = paths.PLUGINS_RESOURCES_PATH / (self.name + '.py')
 180        if potential_file.exists() and not potential_file.is_dir():
 181            return str(potential_file.as_posix())
 182
 183        return None
 184
 185
 186    @property
 187    def requirements_file_path(self) -> Union[pathlib.Path, None]:
 188        """
 189        If a file named `requirements.txt` exists, return its path.
 190        """
 191        if self.__file__ is None:
 192            return None
 193        path = pathlib.Path(self.__file__).parent / 'requirements.txt'
 194        if not path.exists():
 195            return None
 196        return path
 197
 198
 199    def is_installed(self, **kw) -> bool:
 200        """
 201        Check whether a plugin is correctly installed.
 202
 203        Returns
 204        -------
 205        A `bool` indicating whether a plugin exists and is successfully imported.
 206        """
 207        return self.__file__ is not None
 208
 209
 210    def make_tar(self, debug: bool = False) -> pathlib.Path:
 211        """
 212        Compress the plugin's source files into a `.tar.gz` archive and return the archive's path.
 213
 214        Parameters
 215        ----------
 216        debug: bool, default False
 217            Verbosity toggle.
 218
 219        Returns
 220        -------
 221        A `pathlib.Path` to the archive file's path.
 222
 223        """
 224        import tarfile, pathlib, subprocess, fnmatch
 225        from meerschaum.utils.debug import dprint
 226        from meerschaum.utils.packages import attempt_import
 227        pathspec = attempt_import('pathspec', debug=debug)
 228
 229        if not self.__file__:
 230            from meerschaum.utils.warnings import error
 231            error(f"Could not find file for plugin '{self}'.")
 232        if '__init__.py' in self.__file__ or os.path.isdir(self.__file__):
 233            path = self.__file__.replace('__init__.py', '')
 234            is_dir = True
 235        else:
 236            path = self.__file__
 237            is_dir = False
 238
 239        old_cwd = os.getcwd()
 240        real_parent_path = pathlib.Path(os.path.realpath(path)).parent
 241        os.chdir(real_parent_path)
 242
 243        default_patterns_to_ignore = [
 244            '.pyc',
 245            '__pycache__/',
 246            'eggs/',
 247            '__pypackages__/',
 248            '.git',
 249        ]
 250
 251        def parse_gitignore() -> 'Set[str]':
 252            gitignore_path = pathlib.Path(path) / '.gitignore'
 253            if not gitignore_path.exists():
 254                return set(default_patterns_to_ignore)
 255            with open(gitignore_path, 'r', encoding='utf-8') as f:
 256                gitignore_text = f.read()
 257            return set(pathspec.PathSpec.from_lines(
 258                pathspec.patterns.GitWildMatchPattern,
 259                default_patterns_to_ignore + gitignore_text.splitlines()
 260            ).match_tree(path))
 261
 262        patterns_to_ignore = parse_gitignore() if is_dir else set()
 263
 264        if debug:
 265            dprint(f"Patterns to ignore:\n{patterns_to_ignore}")
 266
 267        with tarfile.open(self.archive_path, 'w:gz') as tarf:
 268            if not is_dir:
 269                tarf.add(f"{self.name}.py")
 270            else:
 271                for root, dirs, files in os.walk(self.name):
 272                    for f in files:
 273                        good_file = True
 274                        fp = os.path.join(root, f)
 275                        for pattern in patterns_to_ignore:
 276                            if pattern in str(fp) or f.startswith('.'):
 277                                good_file = False
 278                                break
 279                        if good_file:
 280                            if debug:
 281                                dprint(f"Adding '{fp}'...")
 282                            tarf.add(fp)
 283
 284        ### clean up and change back to old directory
 285        os.chdir(old_cwd)
 286
 287        ### change to 775 to avoid permissions issues with the API in a Docker container
 288        self.archive_path.chmod(0o775)
 289
 290        if debug:
 291            dprint(f"Created archive '{self.archive_path}'.")
 292        return self.archive_path
 293
 294
 295    def install(
 296        self,
 297        skip_deps: bool = False,
 298        force: bool = False,
 299        debug: bool = False,
 300    ) -> SuccessTuple:
 301        """
 302        Extract a plugin's tar archive to the plugins directory.
 303        
 304        This function checks if the plugin is already installed and if the version is equal or
 305        greater than the existing installation.
 306
 307        Parameters
 308        ----------
 309        skip_deps: bool, default False
 310            If `True`, do not install dependencies.
 311
 312        force: bool, default False
 313            If `True`, continue with installation, even if required packages fail to install.
 314
 315        debug: bool, default False
 316            Verbosity toggle.
 317
 318        Returns
 319        -------
 320        A `SuccessTuple` of success (bool) and a message (str).
 321
 322        """
 323        installation_key = self.full_name
 324        with _ongoing_installations_lock:
 325            if installation_key in _ongoing_installations:
 326                return True, f"Already installing plugin '{self}'."
 327            _ongoing_installations.add(installation_key)
 328
 329        try:
 330            from meerschaum.utils.packages import _pip_install_lock
 331            with _pip_install_lock(self.name):
 332                return self._install_with_rollback(
 333                    skip_deps=skip_deps,
 334                    force=force,
 335                    debug=debug,
 336                )
 337        finally:
 338            with _ongoing_installations_lock:
 339                _ongoing_installations.discard(installation_key)
 340
 341
 342    def _install_with_rollback(
 343        self,
 344        skip_deps: bool = False,
 345        force: bool = False,
 346        debug: bool = False,
 347    ) -> SuccessTuple:
 348        """Restore the previous plugin source and environment after a failed install."""
 349        import tempfile
 350        import meerschaum.config.paths as paths
 351        from meerschaum.plugins import sync_plugins_symlinks
 352        from meerschaum.utils.packages import reload_meerschaum
 353
 354        if not self.archive_path.exists():
 355            return False, f"Missing archive file for plugin '{self}'."
 356
 357        paths.PLUGINS_TEMP_RESOURCES_PATH.mkdir(parents=True, exist_ok=True)
 358        source_paths = [
 359            plugins_dir_path / filename
 360            for plugins_dir_path in paths.PLUGINS_DIR_PATHS
 361            for filename in (self.name, self.name + '.py')
 362        ]
 363        tracked_paths = source_paths + [paths.VIRTENV_RESOURCES_PATH / self.name]
 364
 365        with tempfile.TemporaryDirectory(
 366            prefix=f'.{self.name}-rollback-',
 367            dir=paths.PLUGINS_TEMP_RESOURCES_PATH,
 368        ) as rollback_dir_str:
 369            rollback_dir = pathlib.Path(rollback_dir_str)
 370            backups = {}
 371            rollback_prepared = False
 372
 373            def prepare_rollback() -> None:
 374                nonlocal rollback_prepared
 375                for index, path in enumerate(tracked_paths):
 376                    if not path.exists() and not path.is_symlink():
 377                        continue
 378                    backup_path = rollback_dir / str(index)
 379                    _copy_path(path, backup_path)
 380                    backups[path] = backup_path
 381                rollback_prepared = True
 382
 383            def rollback() -> None:
 384                if not rollback_prepared:
 385                    return
 386                for path in tracked_paths:
 387                    _remove_path(path)
 388                for path, backup_path in backups.items():
 389                    path.parent.mkdir(parents=True, exist_ok=True)
 390                    shutil.move(str(backup_path), str(path))
 391                self.__dict__.pop('_module', None)
 392                self.__dict__.pop('_required', None)
 393                sync_plugins_symlinks(debug=debug)
 394                reload_meerschaum(debug=debug)
 395
 396            try:
 397                result = self._install(
 398                    skip_deps=skip_deps,
 399                    force=force,
 400                    debug=debug,
 401                    _prepare_rollback=prepare_rollback,
 402                )
 403            except BaseException:
 404                rollback()
 405                raise
 406            if not result[0]:
 407                try:
 408                    rollback()
 409                except Exception as e:
 410                    return False, result[1] + f"\nFailed to roll back plugin '{self}': {e}"
 411            return result
 412
 413
 414    def _install(
 415        self,
 416        skip_deps: bool = False,
 417        force: bool = False,
 418        debug: bool = False,
 419        _prepare_rollback: Optional[Any] = None,
 420    ) -> SuccessTuple:
 421
 422        import meerschaum.config.paths as paths
 423        from meerschaum.utils.warnings import warn, error
 424        if debug:
 425            from meerschaum.utils.debug import dprint
 426        import tarfile
 427        import re
 428        import ast
 429        from meerschaum.plugins import sync_plugins_symlinks
 430        from meerschaum.utils.packages import attempt_import, reload_meerschaum
 431        from meerschaum.utils.venv import init_venv
 432        from meerschaum.utils.misc import safely_extract_tar
 433        old_cwd = os.getcwd()
 434        old_version = ''
 435        new_version = ''
 436        temp_dir = paths.PLUGINS_TEMP_RESOURCES_PATH / self.name
 437        temp_dir.mkdir(exist_ok=True)
 438
 439        if not self.archive_path.exists():
 440            return False, f"Missing archive file for plugin '{self}'."
 441        if self.version is not None:
 442            old_version = self.version
 443            if debug:
 444                dprint(f"Found existing version '{old_version}' for plugin '{self}'.")
 445
 446        if debug:
 447            dprint(f"Extracting '{self.archive_path}' to '{temp_dir}'...")
 448
 449        try:
 450            with tarfile.open(self.archive_path, 'r:gz') as tarf:
 451                safely_extract_tar(tarf, temp_dir)
 452        except Exception as e:
 453            warn(e)
 454            return False, f"Failed to extract plugin '{self.name}'."
 455
 456        ### search for version information
 457        files = os.listdir(temp_dir)
 458        
 459        if str(files[0]) == self.name:
 460            is_dir = True
 461        elif str(files[0]) == self.name + '.py':
 462            is_dir = False
 463        else:
 464            error(f"Unknown format encountered for plugin '{self}'.")
 465
 466        fpath = temp_dir / files[0]
 467        if is_dir:
 468            fpath = fpath / '__init__.py'
 469
 470        with open(fpath, 'r', encoding='utf-8') as f:
 471            init_lines = f.readlines()
 472        new_version = None
 473        for line in init_lines:
 474            if '__version__' not in line:
 475                continue
 476            version_match = re.search(r'__version__(\s?)=', line.lstrip().rstrip())
 477            if not version_match:
 478                continue
 479            new_version = ast.literal_eval(line.split('=')[1].lstrip().rstrip())
 480            break
 481        if not new_version:
 482            warn(
 483                f"No `__version__` defined for plugin '{self}'. "
 484                + "Assuming new version...",
 485                stack = False,
 486            )
 487
 488        packaging_version = attempt_import('packaging.version')
 489        try:
 490            is_new_version = (not new_version and not old_version) or (
 491                packaging_version.parse(old_version) < packaging_version.parse(new_version)
 492            )
 493            is_same_version = new_version and old_version and (
 494                packaging_version.parse(old_version) == packaging_version.parse(new_version)
 495            )
 496        except Exception:
 497            is_new_version, is_same_version = True, False
 498
 499        ### Determine where to permanently store the new plugin.
 500        plugin_installation_dir_path = paths.PLUGINS_DIR_PATHS[0]
 501        for path in paths.PLUGINS_DIR_PATHS:
 502            if not path.exists():
 503                warn(f"Plugins path does not exist: {path}", stack=False)
 504                continue
 505
 506            files_in_plugins_dir = os.listdir(path)
 507            if (
 508                self.name in files_in_plugins_dir
 509                or
 510                (self.name + '.py') in files_in_plugins_dir
 511            ):
 512                plugin_installation_dir_path = path
 513                break
 514
 515        success_msg = (
 516            f"Successfully installed plugin '{self}'"
 517            + ("\n    (skipped dependencies)" if skip_deps else "")
 518            + "."
 519        )
 520        success, abort = None, None
 521
 522        if is_same_version and not force:
 523            success, msg = True, (
 524                f"Plugin '{self}' is up-to-date (version {old_version}).\n" +
 525                "    Install again with `-f` or `--force` to reinstall."
 526            )
 527            abort = True
 528        elif is_new_version or force:
 529            if _prepare_rollback is not None:
 530                _prepare_rollback()
 531            source_path = temp_dir / files[0]
 532            destination_path = plugin_installation_dir_path / files[0]
 533            try:
 534                plugin_installation_dir_path.mkdir(parents=True, exist_ok=True)
 535                for existing_path in (
 536                    plugin_installation_dir_path / self.name,
 537                    plugin_installation_dir_path / (self.name + '.py'),
 538                ):
 539                    _remove_path(existing_path)
 540                if debug:
 541                    dprint(f"Moving '{source_path}' to '{destination_path}'...")
 542                shutil.move(str(source_path), str(destination_path))
 543            except Exception as e:
 544                success, msg = False, f"Failed to install plugin '{self}': {e}"
 545            else:
 546                success, msg = True, success_msg
 547        else:
 548            success, msg = False, (
 549                f"Your installed version of plugin '{self}' ({old_version}) is higher than "
 550                + f"attempted version {new_version}."
 551            )
 552
 553        shutil.rmtree(temp_dir)
 554        os.chdir(old_cwd)
 555
 556        ### Reload the plugin's module.
 557        sync_plugins_symlinks(debug=debug)
 558        if '_module' in self.__dict__:
 559            del self.__dict__['_module']
 560        init_venv(venv=self.name, force=True, debug=debug)
 561        reload_meerschaum(debug=debug)
 562
 563        ### Record the origin only after a successful transaction. Locally-installed
 564        ### plugins have no explicit repo and are intentionally skipped.
 565        _origin_repo_keys = (
 566            str(self._repo_connector)
 567            if self._repo_connector is not None
 568            else self._repo_keys or self._repo_in_name
 569        )
 570
 571        def write_origin() -> None:
 572            if not _origin_repo_keys:
 573                return
 574            try:
 575                from meerschaum.plugins._origins import write_plugin_origin
 576                write_plugin_origin(
 577                    self.name,
 578                    _origin_repo_keys,
 579                    plugin_installation_dir_path,
 580                    debug=debug,
 581                )
 582            except Exception:
 583                pass
 584
 585        if abort:
 586            write_origin()
 587
 588        ### if we've already failed, return here
 589        if not success or abort:
 590            return success, msg
 591
 592        ### attempt to install dependencies
 593        dependencies_installed = skip_deps or self.install_dependencies(force=force, debug=debug)
 594        if not dependencies_installed:
 595            return False, f"Failed to install dependencies for plugin '{self}'."
 596
 597        ### handling success tuple, bool, or other (typically None)
 598        setup_tuple = self.setup(debug=debug)
 599        if isinstance(setup_tuple, tuple):
 600            if not setup_tuple[0]:
 601                success, msg = setup_tuple
 602        elif isinstance(setup_tuple, bool):
 603            if not setup_tuple:
 604                success, msg = False, (
 605                    f"Failed to run post-install setup for plugin '{self}'." + '\n' +
 606                    f"Check `setup()` in '{self.__file__}' for more information " +
 607                    "(no error message provided)."
 608                )
 609            else:
 610                success, msg = True, success_msg
 611        elif setup_tuple is None:
 612            success = True
 613            msg = (
 614                f"Post-install for plugin '{self}' returned None. " +
 615                "Assuming plugin successfully installed."
 616            )
 617            warn(msg)
 618        else:
 619            success = False
 620            msg = (
 621                f"Post-install for plugin '{self}' returned unexpected value " +
 622                f"of type '{type(setup_tuple)}': {setup_tuple}"
 623            )
 624
 625        _ = self.module
 626        if success:
 627            write_origin()
 628        return success, msg
 629
 630
 631    def remove_archive(
 632        self,        
 633        debug: bool = False
 634    ) -> SuccessTuple:
 635        """Remove a plugin's archive file."""
 636        if not self.archive_path.exists():
 637            return True, f"Archive file for plugin '{self}' does not exist."
 638        try:
 639            self.archive_path.unlink()
 640        except Exception as e:
 641            return False, f"Failed to remove archive for plugin '{self}':\n{e}"
 642        return True, "Success"
 643
 644
 645    def remove_venv(
 646        self,        
 647        debug: bool = False
 648    ) -> SuccessTuple:
 649        """Remove a plugin's virtual environment."""
 650        if not self.venv_path.exists():
 651            return True, f"Virtual environment for plugin '{self}' does not exist."
 652        try:
 653            shutil.rmtree(self.venv_path)
 654        except Exception as e:
 655            return False, f"Failed to remove virtual environment for plugin '{self}':\n{e}"
 656        return True, "Success"
 657
 658
 659    def uninstall(self, debug: bool = False) -> SuccessTuple:
 660        """
 661        Remove a plugin, its virtual environment, and archive file.
 662        """
 663        from meerschaum.utils.packages import reload_meerschaum
 664        from meerschaum.plugins import sync_plugins_symlinks
 665        from meerschaum.utils.warnings import warn, info
 666        warnings_thrown_count: int = 0
 667        max_warnings: int = 3
 668
 669        if not self.is_installed():
 670            info(
 671                f"Plugin '{self.name}' doesn't seem to be installed.\n    "
 672                + "Checking for artifacts...",
 673                stack = False,
 674            )
 675        else:
 676            real_path = pathlib.Path(os.path.realpath(self.__file__))
 677            try:
 678                if real_path.name == '__init__.py':
 679                    shutil.rmtree(real_path.parent)
 680                else:
 681                    real_path.unlink()
 682            except Exception as e:
 683                warn(f"Could not remove source files for plugin '{self.name}':\n{e}", stack=False)
 684                warnings_thrown_count += 1
 685            else:
 686                info(f"Removed source files for plugin '{self.name}'.")
 687
 688        if self.venv_path.exists():
 689            success, msg = self.remove_venv(debug=debug)
 690            if not success:
 691                warn(msg, stack=False)
 692                warnings_thrown_count += 1
 693            else:
 694                info(f"Removed virtual environment from plugin '{self.name}'.")
 695
 696        success = warnings_thrown_count < max_warnings
 697        try:
 698            from meerschaum.plugins._origins import remove_plugin_origin
 699            remove_plugin_origin(self.name, debug=debug)
 700        except Exception:
 701            pass
 702        sync_plugins_symlinks(debug=debug)
 703        self.deactivate_venv(force=True, debug=debug)
 704        reload_meerschaum(debug=debug)
 705        return success, (
 706            f"Successfully uninstalled plugin '{self}'." if success
 707            else f"Failed to uninstall plugin '{self}'."
 708        )
 709
 710
 711    def setup(self, *args: str, debug: bool = False, **kw: Any) -> Union[SuccessTuple, bool]:
 712        """
 713        If exists, run the plugin's `setup()` function.
 714
 715        Parameters
 716        ----------
 717        *args: str
 718            The positional arguments passed to the `setup()` function.
 719            
 720        debug: bool, default False
 721            Verbosity toggle.
 722
 723        **kw: Any
 724            The keyword arguments passed to the `setup()` function.
 725
 726        Returns
 727        -------
 728        A `SuccessTuple` or `bool` indicating success.
 729
 730        """
 731        from meerschaum.utils.debug import dprint
 732        import inspect
 733        _setup = None
 734        for name, fp in inspect.getmembers(self.module):
 735            if name == 'setup' and inspect.isfunction(fp):
 736                _setup = fp
 737                break
 738
 739        ### assume success if no setup() is found (not necessary)
 740        if _setup is None:
 741            return True
 742
 743        sig = inspect.signature(_setup)
 744        has_debug, has_kw = ('debug' in sig.parameters), False
 745        for k, v in sig.parameters.items():
 746            if '**' in str(v):
 747                has_kw = True
 748                break
 749
 750        _kw = {}
 751        if has_kw:
 752            _kw.update(kw)
 753        if has_debug:
 754            _kw['debug'] = debug
 755
 756        if debug:
 757            dprint(f"Running setup for plugin '{self}'...")
 758        from meerschaum.utils.venv import Venv
 759        try:
 760            with Venv(self, debug=debug):
 761                return_tuple = _setup(*args, **_kw)
 762        except Exception as e:
 763            return False, str(e)
 764
 765        if isinstance(return_tuple, tuple):
 766            return return_tuple
 767        if isinstance(return_tuple, bool):
 768            return return_tuple, f"Setup for Plugin '{self.name}' did not return a message."
 769        if return_tuple is None:
 770            return False, f"Setup for Plugin '{self.name}' returned None."
 771        return False, f"Unknown return value from setup for Plugin '{self.name}': {return_tuple}"
 772
 773
 774    def get_dependencies(
 775        self,
 776        debug: bool = False,
 777    ) -> List[str]:
 778        """
 779        If the Plugin has specified dependencies in a list called `required`, return the list.
 780        
 781        **NOTE:** Dependecies which start with `'plugin:'` are Meerschaum plugins, not pip packages.
 782        Meerschaum plugins may also specify connector keys for a repo after `'@'`.
 783
 784        Parameters
 785        ----------
 786        debug: bool, default False
 787            Verbosity toggle.
 788
 789        Returns
 790        -------
 791        A list of required packages and plugins (str).
 792
 793        """
 794        if '_required' in self.__dict__:
 795            return self._required
 796
 797        ### If the plugin has not yet been imported,
 798        ### infer the dependencies from the source text.
 799        ### This is not super robust, and it doesn't feel right
 800        ### having multiple versions of the logic.
 801        ### This is necessary when determining the activation order
 802        ### without having import the module.
 803        ### For consistency's sake, the module-less method does not cache the requirements.
 804        if self.__dict__.get('_module', None) is None:
 805            file_path = self.__file__
 806            if file_path is None:
 807                return []
 808            with open(file_path, 'r', encoding='utf-8') as f:
 809                text = f.read()
 810
 811            if 'required' not in text:
 812                return []
 813
 814            ### This has some limitations:
 815            ### It relies on `required` being manually declared.
 816            ### We lose the ability to dynamically alter the `required` list,
 817            ### which is why we've kept the module-reliant method below.
 818            import ast, re
 819            ### NOTE: This technically would break 
 820            ### if `required` was the very first line of the file.
 821            req_start_match = re.search(r'\nrequired(:\s*)?.*=', text)
 822            if not req_start_match:
 823                return []
 824            req_start = req_start_match.start()
 825            equals_sign = req_start + text[req_start:].find('=')
 826
 827            ### Dependencies may have brackets within the strings, so push back the index.
 828            first_opening_brace = equals_sign + 1 + text[equals_sign:].find('[')
 829            if first_opening_brace == -1:
 830                return []
 831
 832            next_closing_brace = equals_sign + 1 + text[equals_sign:].find(']')
 833            if next_closing_brace == -1:
 834                return []
 835
 836            start_ix = first_opening_brace + 1
 837            end_ix = next_closing_brace
 838
 839            num_braces = 0
 840            while True:
 841                if '[' not in text[start_ix:end_ix]:
 842                    break
 843                num_braces += 1
 844                start_ix = end_ix
 845                end_ix += text[end_ix + 1:].find(']') + 1
 846
 847            req_end = end_ix + 1
 848            req_text = (
 849                text[(first_opening_brace-1):req_end]
 850                .lstrip()
 851                .replace('=', '', 1)
 852                .lstrip()
 853                .rstrip()
 854            )
 855            try:
 856                required = ast.literal_eval(req_text)
 857            except Exception as e:
 858                warn(
 859                    f"Unable to determine requirements for plugin '{self.name}' "
 860                    + "without importing the module.\n"
 861                    + "    This may be due to dynamically setting the global `required` list.\n"
 862                    + f"    {e}"
 863                )
 864                return []
 865            return required
 866
 867        import inspect
 868        self.activate_venv(dependencies=False, debug=debug)
 869        required = []
 870        for name, val in inspect.getmembers(self.module):
 871            if name == 'required':
 872                required = val
 873                break
 874        self._required = required
 875        self.deactivate_venv(dependencies=False, debug=debug)
 876        return required
 877
 878
 879    def get_required_plugins(self, debug: bool=False) -> List[mrsm.plugins.Plugin]:
 880        """
 881        Return a list of required Plugin objects.
 882        """
 883        from meerschaum.utils.warnings import warn
 884        from meerschaum.config import get_config
 885        from meerschaum._internal.static import STATIC_CONFIG
 886        from meerschaum.connectors.parse import is_valid_connector_keys
 887        plugins = []
 888        _deps = self.get_dependencies(debug=debug)
 889        sep = STATIC_CONFIG['plugins']['repo_separator']
 890        plugin_names = [
 891            _d[len('plugin:'):] for _d in _deps
 892            if _d.startswith('plugin:') and len(_d) > len('plugin:')
 893        ]
 894        default_repo_keys = get_config('meerschaum', 'repository')
 895        skipped_repo_keys = set()
 896
 897        for _plugin_name in plugin_names:
 898            if sep in _plugin_name:
 899                try:
 900                    _plugin_name, _repo_keys = _plugin_name.split(sep)
 901                except Exception:
 902                    _repo_keys = default_repo_keys
 903                    warn(
 904                        f"Invalid repo keys for required plugin '{_plugin_name}'.\n    "
 905                        + f"Will try to use '{_repo_keys}' instead.",
 906                        stack = False,
 907                    )
 908            else:
 909                _repo_keys = default_repo_keys
 910
 911            if _repo_keys in skipped_repo_keys:
 912                continue
 913
 914            if not is_valid_connector_keys(_repo_keys):
 915                warn(
 916                    f"Invalid connector '{_repo_keys}'.\n"
 917                    f"    Skipping required plugins from repository '{_repo_keys}'",
 918                    stack=False,
 919                )
 920                continue
 921
 922            plugins.append(Plugin(_plugin_name, repo=_repo_keys))
 923
 924        return plugins
 925
 926
 927    def get_required_packages(self, debug: bool=False) -> List[str]:
 928        """
 929        Return the required package names (excluding plugins).
 930        """
 931        _deps = self.get_dependencies(debug=debug)
 932        return [_d for _d in _deps if not _d.startswith('plugin:')]
 933
 934
 935    def activate_venv(
 936        self,
 937        dependencies: bool = True,
 938        init_if_not_exists: bool = True,
 939        debug: bool = False,
 940        **kw
 941    ) -> bool:
 942        """
 943        Activate the virtual environments for the plugin and its dependencies.
 944
 945        Parameters
 946        ----------
 947        dependencies: bool, default True
 948            If `True`, activate the virtual environments for required plugins.
 949
 950        Returns
 951        -------
 952        A bool indicating success.
 953        """
 954        import meerschaum.config.paths as paths
 955        from meerschaum.utils.venv import venv_target_path
 956        from meerschaum.utils.packages import activate_venv
 957        from meerschaum.utils.misc import make_symlink, is_symlink
 958
 959        if dependencies:
 960            for plugin in self.get_required_plugins(debug=debug):
 961                plugin.activate_venv(debug=debug, init_if_not_exists=init_if_not_exists, **kw)
 962
 963        vtp = venv_target_path(self.name, debug=debug, allow_nonexistent=True)
 964        venv_meerschaum_path = vtp / 'meerschaum'
 965
 966        try:
 967            success, msg = True, "Success"
 968            if is_symlink(venv_meerschaum_path):
 969                if pathlib.Path(os.path.realpath(venv_meerschaum_path)) != paths.PACKAGE_ROOT_PATH:
 970                    venv_meerschaum_path.unlink()
 971                    success, msg = make_symlink(venv_meerschaum_path, paths.PACKAGE_ROOT_PATH)
 972        except Exception as e:
 973            success, msg = False, str(e)
 974        if not success:
 975            warn(
 976                f"Unable to create symlink {venv_meerschaum_path} to {paths.PACKAGE_ROOT_PATH}:\n"
 977                f"{msg}"
 978            )
 979
 980        return activate_venv(self.name, init_if_not_exists=init_if_not_exists, debug=debug, **kw)
 981
 982
 983    def deactivate_venv(self, dependencies: bool=True, debug: bool = False, **kw) -> bool:
 984        """
 985        Deactivate the virtual environments for the plugin and its dependencies.
 986
 987        Parameters
 988        ----------
 989        dependencies: bool, default True
 990            If `True`, deactivate the virtual environments for required plugins.
 991
 992        Returns
 993        -------
 994        A bool indicating success.
 995        """
 996        from meerschaum.utils.packages import deactivate_venv
 997        success = deactivate_venv(self.name, debug=debug, **kw)
 998        if dependencies:
 999            for plugin in self.get_required_plugins(debug=debug):
1000                plugin.deactivate_venv(debug=debug, **kw)
1001        return success
1002
1003
1004    def install_dependencies(
1005        self,
1006        force: bool = False,
1007        debug: bool = False,
1008    ) -> bool:
1009        """
1010        If specified, install dependencies.
1011        
1012        **NOTE:** Dependencies that start with `'plugin:'` will be installed as
1013        Meerschaum plugins from the same repository as this Plugin.
1014        To install from a different repository, add the repo keys after `'@'`
1015        (e.g. `'plugin:foo@api:bar'`).
1016
1017        Parameters
1018        ----------
1019        force: bool, default False
1020            If `True`, continue with the installation, even if some
1021            required packages fail to install.
1022
1023        debug: bool, default False
1024            Verbosity toggle.
1025
1026        Returns
1027        -------
1028        A bool indicating success.
1029        """
1030        from meerschaum.utils.packages import pip_install, venv_contains_package
1031        from meerschaum.utils.warnings import warn, info
1032        _deps = self.get_dependencies(debug=debug)
1033        if not _deps and self.requirements_file_path is None:
1034            return True
1035
1036        plugins = self.get_required_plugins(debug=debug)
1037        for _plugin in plugins:
1038            if _plugin.name == self.name:
1039                warn(f"Plugin '{self.name}' cannot depend on itself! Skipping...", stack=False)
1040                continue
1041            _success, _msg = _plugin.repo_connector.install_plugin(
1042                _plugin.name, debug=debug, force=force
1043            )
1044            if not _success:
1045                warn(
1046                    f"Failed to install required plugin '{_plugin}' from '{_plugin.repo_connector}'"
1047                    + f" for plugin '{self.name}':\n" + _msg,
1048                    stack = False,
1049                )
1050                if not force:
1051                    warn(
1052                        "Try installing with the `--force` flag to continue anyway.",
1053                        stack = False,
1054                    )
1055                    return False
1056                info(
1057                    "Continuing with installation despite the failure "
1058                    + "(careful, things might be broken!)...",
1059                    icon = False
1060                )
1061
1062
1063        ### First step: parse `requirements.txt` if it exists.
1064        if self.requirements_file_path is not None:
1065            if not pip_install(
1066                requirements_file_path=self.requirements_file_path,
1067                venv=self.name, debug=debug
1068            ):
1069                warn(
1070                    f"Failed to resolve 'requirements.txt' for plugin '{self.name}'.",
1071                    stack = False,
1072                )
1073                if not force:
1074                    warn(
1075                        "Try installing with `--force` to continue anyway.",
1076                        stack = False,
1077                    )
1078                    return False
1079                info(
1080                    "Continuing with installation despite the failure "
1081                    + "(careful, things might be broken!)...",
1082                    icon = False
1083                )
1084
1085
1086        ### Don't reinstall packages that are already included in required plugins.
1087        packages = []
1088        _packages = self.get_required_packages(debug=debug)
1089        accounted_for_packages = set()
1090        for package_name in _packages:
1091            for plugin in plugins:
1092                if venv_contains_package(package_name, plugin.name):
1093                    accounted_for_packages.add(package_name)
1094                    break
1095        packages = [pkg for pkg in _packages if pkg not in accounted_for_packages]
1096
1097        ### Attempt pip packages installation.
1098        if packages:
1099            for package in packages:
1100                if not pip_install(package, venv=self.name, debug=debug):
1101                    warn(
1102                        f"Failed to install required package '{package}'"
1103                        + f" for plugin '{self.name}'.",
1104                        stack = False,
1105                    )
1106                    if not force:
1107                        warn(
1108                            "Try installing with `--force` to continue anyway.",
1109                            stack = False,
1110                        )
1111                        return False
1112                    info(
1113                        "Continuing with installation despite the failure "
1114                        + "(careful, things might be broken!)...",
1115                        icon = False
1116                    )
1117        return True
1118
1119
1120    @property
1121    def full_name(self) -> str:
1122        """
1123        Include the repo keys with the plugin's name.
1124        """
1125        from meerschaum._internal.static import STATIC_CONFIG
1126        sep = STATIC_CONFIG['plugins']['repo_separator']
1127        return self.name + sep + str(self.repo_connector)
1128
1129
1130    def __str__(self):
1131        return self.name
1132
1133
1134    def __repr__(self):
1135        return f"Plugin('{self.name}', repo='{self.repo_connector}')"
1136
1137
1138    def __del__(self):
1139        pass

Handle packaging of Meerschaum plugins.

Plugin( name: str, version: str | None = None, user_id: int | None = None, required: List[str] | None = None, attributes: Dict[str, Any] | None = None, archive_path: pathlib.Path | None = None, venv_path: pathlib.Path | None = None, repo_connector: meerschaum.connectors.APIConnector | None = None, repo: meerschaum.connectors.APIConnector | str | None = None) 
53    def __init__(
54        self,
55        name: str,
56        version: Optional[str] = None,
57        user_id: Optional[int] = None,
58        required: Optional[List[str]] = None,
59        attributes: Optional[Dict[str, Any]] = None,
60        archive_path: Optional[pathlib.Path] = None,
61        venv_path: Optional[pathlib.Path] = None,
62        repo_connector: Optional['mrsm.connectors.api.APIConnector'] = None,
63        repo: Union['mrsm.connectors.api.APIConnector', str, None] = None,
64    ):
65        import meerschaum.config.paths as paths
66        from meerschaum._internal.static import STATIC_CONFIG
67        sep = STATIC_CONFIG['plugins']['repo_separator']
68        _repo = None
69        if sep in name:
70            try:
71                name, _repo = name.split(sep)
72            except Exception as e:
73                error(f"Invalid plugin name: '{name}'")
74        self._repo_in_name = _repo
75
76        if attributes is None:
77            attributes = {}
78        self.name = name
79        self.attributes = attributes
80        self.user_id = user_id
81        self._version = version
82        if required:
83            self._required = required
84        self.archive_path = (
85            archive_path if archive_path is not None
86            else paths.PLUGINS_ARCHIVES_RESOURCES_PATH / f"{self.name}.tar.gz"
87        )
88        self.venv_path = (
89            venv_path if venv_path is not None
90            else paths.VIRTENV_RESOURCES_PATH / self.name
91        )
92        self._repo_connector = repo_connector
93        self._repo_keys = repo
name
attributes
user_id
archive_path
venv_path
repo_connector 
 96    @property
 97    def repo_connector(self):
 98        """
 99        Return the repository connector for this plugin.
100        NOTE: This imports the `connectors` module, which imports certain plugin modules.
101        """
102        if self._repo_connector is None:
103            from meerschaum.connectors.parse import parse_repo_keys
104
105            repo_keys = self._repo_keys or self._repo_in_name
106            if self._repo_in_name and self._repo_keys and self._repo_keys != self._repo_in_name:
107                error(
108                    f"Received inconsistent repos: '{self._repo_in_name}' and '{self._repo_keys}'."
109                )
110            repo_connector = parse_repo_keys(repo_keys)
111            self._repo_connector = repo_connector
112        return self._repo_connector

Return the repository connector for this plugin. NOTE: This imports the connectors module, which imports certain plugin modules.

version 
115    @property
116    def version(self):
117        """
118        Return the plugin's module version is defined (`__version__`) if it's defined.
119        """
120        if self._version is None:
121            try:
122                self._version = self.module.__version__
123            except Exception as e:
124                self._version = None
125        return self._version

Return the plugin's module version is defined (__version__) if it's defined.

module 
128    @property
129    def module(self):
130        """
131        Return the Python module of the underlying plugin,
132        or `None` if the plugin is not installed or its import failed
133        (check `Plugin.import_error` for the swallowed exception).
134        """
135        if '_module' not in self.__dict__ or self.__dict__.get('_module', None) is None:
136            from meerschaum.plugins import _get_loaded_module, _get_plugins_package_name
137            loaded_module = _get_loaded_module(f'{_get_plugins_package_name()}.{self.name}')
138            if loaded_module is not None:
139                self._module = loaded_module
140                return self._module
141
142            if self.__file__ is None:
143                return None
144
145            from meerschaum.plugins import import_plugins
146            self._module = import_plugins(str(self), warn=False)
147
148        return self._module

Return the Python module of the underlying plugin, or None if the plugin is not installed or its import failed (check Plugin.import_error for the swallowed exception).

import_error: Exception | None 
150    @property
151    def import_error(self) -> Union[Exception, None]:
152        """
153        Return the exception raised by this plugin's most recent failed import
154        (i.e. when `Plugin.module` is `None` even though the plugin is installed),
155        otherwise `None`.
156        """
157        from meerschaum.plugins import _plugins_import_errors
158        return _plugins_import_errors.get(self.name, None)

Return the exception raised by this plugin's most recent failed import (i.e. when Plugin.module is None even though the plugin is installed), otherwise None.

requirements_file_path: pathlib.Path | None 
186    @property
187    def requirements_file_path(self) -> Union[pathlib.Path, None]:
188        """
189        If a file named `requirements.txt` exists, return its path.
190        """
191        if self.__file__ is None:
192            return None
193        path = pathlib.Path(self.__file__).parent / 'requirements.txt'
194        if not path.exists():
195            return None
196        return path

If a file named requirements.txt exists, return its path.

def is_installed(self, **kw) -> bool: 
199    def is_installed(self, **kw) -> bool:
200        """
201        Check whether a plugin is correctly installed.
202
203        Returns
204        -------
205        A `bool` indicating whether a plugin exists and is successfully imported.
206        """
207        return self.__file__ is not None

Check whether a plugin is correctly installed.

Returns
  • A bool indicating whether a plugin exists and is successfully imported.
def make_tar(self, debug: bool = False) -> pathlib.Path: 
210    def make_tar(self, debug: bool = False) -> pathlib.Path:
211        """
212        Compress the plugin's source files into a `.tar.gz` archive and return the archive's path.
213
214        Parameters
215        ----------
216        debug: bool, default False
217            Verbosity toggle.
218
219        Returns
220        -------
221        A `pathlib.Path` to the archive file's path.
222
223        """
224        import tarfile, pathlib, subprocess, fnmatch
225        from meerschaum.utils.debug import dprint
226        from meerschaum.utils.packages import attempt_import
227        pathspec = attempt_import('pathspec', debug=debug)
228
229        if not self.__file__:
230            from meerschaum.utils.warnings import error
231            error(f"Could not find file for plugin '{self}'.")
232        if '__init__.py' in self.__file__ or os.path.isdir(self.__file__):
233            path = self.__file__.replace('__init__.py', '')
234            is_dir = True
235        else:
236            path = self.__file__
237            is_dir = False
238
239        old_cwd = os.getcwd()
240        real_parent_path = pathlib.Path(os.path.realpath(path)).parent
241        os.chdir(real_parent_path)
242
243        default_patterns_to_ignore = [
244            '.pyc',
245            '__pycache__/',
246            'eggs/',
247            '__pypackages__/',
248            '.git',
249        ]
250
251        def parse_gitignore() -> 'Set[str]':
252            gitignore_path = pathlib.Path(path) / '.gitignore'
253            if not gitignore_path.exists():
254                return set(default_patterns_to_ignore)
255            with open(gitignore_path, 'r', encoding='utf-8') as f:
256                gitignore_text = f.read()
257            return set(pathspec.PathSpec.from_lines(
258                pathspec.patterns.GitWildMatchPattern,
259                default_patterns_to_ignore + gitignore_text.splitlines()
260            ).match_tree(path))
261
262        patterns_to_ignore = parse_gitignore() if is_dir else set()
263
264        if debug:
265            dprint(f"Patterns to ignore:\n{patterns_to_ignore}")
266
267        with tarfile.open(self.archive_path, 'w:gz') as tarf:
268            if not is_dir:
269                tarf.add(f"{self.name}.py")
270            else:
271                for root, dirs, files in os.walk(self.name):
272                    for f in files:
273                        good_file = True
274                        fp = os.path.join(root, f)
275                        for pattern in patterns_to_ignore:
276                            if pattern in str(fp) or f.startswith('.'):
277                                good_file = False
278                                break
279                        if good_file:
280                            if debug:
281                                dprint(f"Adding '{fp}'...")
282                            tarf.add(fp)
283
284        ### clean up and change back to old directory
285        os.chdir(old_cwd)
286
287        ### change to 775 to avoid permissions issues with the API in a Docker container
288        self.archive_path.chmod(0o775)
289
290        if debug:
291            dprint(f"Created archive '{self.archive_path}'.")
292        return self.archive_path

Compress the plugin's source files into a .tar.gz archive and return the archive's path.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
  • A pathlib.Path to the archive file's path.
def install( self, skip_deps: bool = False, force: bool = False, debug: bool = False) -> Tuple[bool, str]: 
295    def install(
296        self,
297        skip_deps: bool = False,
298        force: bool = False,
299        debug: bool = False,
300    ) -> SuccessTuple:
301        """
302        Extract a plugin's tar archive to the plugins directory.
303        
304        This function checks if the plugin is already installed and if the version is equal or
305        greater than the existing installation.
306
307        Parameters
308        ----------
309        skip_deps: bool, default False
310            If `True`, do not install dependencies.
311
312        force: bool, default False
313            If `True`, continue with installation, even if required packages fail to install.
314
315        debug: bool, default False
316            Verbosity toggle.
317
318        Returns
319        -------
320        A `SuccessTuple` of success (bool) and a message (str).
321
322        """
323        installation_key = self.full_name
324        with _ongoing_installations_lock:
325            if installation_key in _ongoing_installations:
326                return True, f"Already installing plugin '{self}'."
327            _ongoing_installations.add(installation_key)
328
329        try:
330            from meerschaum.utils.packages import _pip_install_lock
331            with _pip_install_lock(self.name):
332                return self._install_with_rollback(
333                    skip_deps=skip_deps,
334                    force=force,
335                    debug=debug,
336                )
337        finally:
338            with _ongoing_installations_lock:
339                _ongoing_installations.discard(installation_key)

Extract a plugin's tar archive to the plugins directory.

This function checks if the plugin is already installed and if the version is equal or greater than the existing installation.

Parameters
  • skip_deps (bool, default False): If True, do not install dependencies.
  • force (bool, default False): If True, continue with installation, even if required packages fail to install.
  • debug (bool, default False): Verbosity toggle.
Returns
def remove_archive(self, debug: bool = False) -> Tuple[bool, str]: 
631    def remove_archive(
632        self,        
633        debug: bool = False
634    ) -> SuccessTuple:
635        """Remove a plugin's archive file."""
636        if not self.archive_path.exists():
637            return True, f"Archive file for plugin '{self}' does not exist."
638        try:
639            self.archive_path.unlink()
640        except Exception as e:
641            return False, f"Failed to remove archive for plugin '{self}':\n{e}"
642        return True, "Success"

Remove a plugin's archive file.

def remove_venv(self, debug: bool = False) -> Tuple[bool, str]: 
645    def remove_venv(
646        self,        
647        debug: bool = False
648    ) -> SuccessTuple:
649        """Remove a plugin's virtual environment."""
650        if not self.venv_path.exists():
651            return True, f"Virtual environment for plugin '{self}' does not exist."
652        try:
653            shutil.rmtree(self.venv_path)
654        except Exception as e:
655            return False, f"Failed to remove virtual environment for plugin '{self}':\n{e}"
656        return True, "Success"

Remove a plugin's virtual environment.

def uninstall(self, debug: bool = False) -> Tuple[bool, str]: 
659    def uninstall(self, debug: bool = False) -> SuccessTuple:
660        """
661        Remove a plugin, its virtual environment, and archive file.
662        """
663        from meerschaum.utils.packages import reload_meerschaum
664        from meerschaum.plugins import sync_plugins_symlinks
665        from meerschaum.utils.warnings import warn, info
666        warnings_thrown_count: int = 0
667        max_warnings: int = 3
668
669        if not self.is_installed():
670            info(
671                f"Plugin '{self.name}' doesn't seem to be installed.\n    "
672                + "Checking for artifacts...",
673                stack = False,
674            )
675        else:
676            real_path = pathlib.Path(os.path.realpath(self.__file__))
677            try:
678                if real_path.name == '__init__.py':
679                    shutil.rmtree(real_path.parent)
680                else:
681                    real_path.unlink()
682            except Exception as e:
683                warn(f"Could not remove source files for plugin '{self.name}':\n{e}", stack=False)
684                warnings_thrown_count += 1
685            else:
686                info(f"Removed source files for plugin '{self.name}'.")
687
688        if self.venv_path.exists():
689            success, msg = self.remove_venv(debug=debug)
690            if not success:
691                warn(msg, stack=False)
692                warnings_thrown_count += 1
693            else:
694                info(f"Removed virtual environment from plugin '{self.name}'.")
695
696        success = warnings_thrown_count < max_warnings
697        try:
698            from meerschaum.plugins._origins import remove_plugin_origin
699            remove_plugin_origin(self.name, debug=debug)
700        except Exception:
701            pass
702        sync_plugins_symlinks(debug=debug)
703        self.deactivate_venv(force=True, debug=debug)
704        reload_meerschaum(debug=debug)
705        return success, (
706            f"Successfully uninstalled plugin '{self}'." if success
707            else f"Failed to uninstall plugin '{self}'."
708        )

Remove a plugin, its virtual environment, and archive file.

def setup( self, *args: str, debug: bool = False, **kw: Any) -> Tuple[bool, str] | bool: 
711    def setup(self, *args: str, debug: bool = False, **kw: Any) -> Union[SuccessTuple, bool]:
712        """
713        If exists, run the plugin's `setup()` function.
714
715        Parameters
716        ----------
717        *args: str
718            The positional arguments passed to the `setup()` function.
719            
720        debug: bool, default False
721            Verbosity toggle.
722
723        **kw: Any
724            The keyword arguments passed to the `setup()` function.
725
726        Returns
727        -------
728        A `SuccessTuple` or `bool` indicating success.
729
730        """
731        from meerschaum.utils.debug import dprint
732        import inspect
733        _setup = None
734        for name, fp in inspect.getmembers(self.module):
735            if name == 'setup' and inspect.isfunction(fp):
736                _setup = fp
737                break
738
739        ### assume success if no setup() is found (not necessary)
740        if _setup is None:
741            return True
742
743        sig = inspect.signature(_setup)
744        has_debug, has_kw = ('debug' in sig.parameters), False
745        for k, v in sig.parameters.items():
746            if '**' in str(v):
747                has_kw = True
748                break
749
750        _kw = {}
751        if has_kw:
752            _kw.update(kw)
753        if has_debug:
754            _kw['debug'] = debug
755
756        if debug:
757            dprint(f"Running setup for plugin '{self}'...")
758        from meerschaum.utils.venv import Venv
759        try:
760            with Venv(self, debug=debug):
761                return_tuple = _setup(*args, **_kw)
762        except Exception as e:
763            return False, str(e)
764
765        if isinstance(return_tuple, tuple):
766            return return_tuple
767        if isinstance(return_tuple, bool):
768            return return_tuple, f"Setup for Plugin '{self.name}' did not return a message."
769        if return_tuple is None:
770            return False, f"Setup for Plugin '{self.name}' returned None."
771        return False, f"Unknown return value from setup for Plugin '{self.name}': {return_tuple}"

If exists, run the plugin's setup() function.

Parameters
  • *args (str): The positional arguments passed to the setup() function.
  • debug (bool, default False): Verbosity toggle.
  • **kw (Any): The keyword arguments passed to the setup() function.
Returns
def get_dependencies(self, debug: bool = False) -> List[str]: 
774    def get_dependencies(
775        self,
776        debug: bool = False,
777    ) -> List[str]:
778        """
779        If the Plugin has specified dependencies in a list called `required`, return the list.
780        
781        **NOTE:** Dependecies which start with `'plugin:'` are Meerschaum plugins, not pip packages.
782        Meerschaum plugins may also specify connector keys for a repo after `'@'`.
783
784        Parameters
785        ----------
786        debug: bool, default False
787            Verbosity toggle.
788
789        Returns
790        -------
791        A list of required packages and plugins (str).
792
793        """
794        if '_required' in self.__dict__:
795            return self._required
796
797        ### If the plugin has not yet been imported,
798        ### infer the dependencies from the source text.
799        ### This is not super robust, and it doesn't feel right
800        ### having multiple versions of the logic.
801        ### This is necessary when determining the activation order
802        ### without having import the module.
803        ### For consistency's sake, the module-less method does not cache the requirements.
804        if self.__dict__.get('_module', None) is None:
805            file_path = self.__file__
806            if file_path is None:
807                return []
808            with open(file_path, 'r', encoding='utf-8') as f:
809                text = f.read()
810
811            if 'required' not in text:
812                return []
813
814            ### This has some limitations:
815            ### It relies on `required` being manually declared.
816            ### We lose the ability to dynamically alter the `required` list,
817            ### which is why we've kept the module-reliant method below.
818            import ast, re
819            ### NOTE: This technically would break 
820            ### if `required` was the very first line of the file.
821            req_start_match = re.search(r'\nrequired(:\s*)?.*=', text)
822            if not req_start_match:
823                return []
824            req_start = req_start_match.start()
825            equals_sign = req_start + text[req_start:].find('=')
826
827            ### Dependencies may have brackets within the strings, so push back the index.
828            first_opening_brace = equals_sign + 1 + text[equals_sign:].find('[')
829            if first_opening_brace == -1:
830                return []
831
832            next_closing_brace = equals_sign + 1 + text[equals_sign:].find(']')
833            if next_closing_brace == -1:
834                return []
835
836            start_ix = first_opening_brace + 1
837            end_ix = next_closing_brace
838
839            num_braces = 0
840            while True:
841                if '[' not in text[start_ix:end_ix]:
842                    break
843                num_braces += 1
844                start_ix = end_ix
845                end_ix += text[end_ix + 1:].find(']') + 1
846
847            req_end = end_ix + 1
848            req_text = (
849                text[(first_opening_brace-1):req_end]
850                .lstrip()
851                .replace('=', '', 1)
852                .lstrip()
853                .rstrip()
854            )
855            try:
856                required = ast.literal_eval(req_text)
857            except Exception as e:
858                warn(
859                    f"Unable to determine requirements for plugin '{self.name}' "
860                    + "without importing the module.\n"
861                    + "    This may be due to dynamically setting the global `required` list.\n"
862                    + f"    {e}"
863                )
864                return []
865            return required
866
867        import inspect
868        self.activate_venv(dependencies=False, debug=debug)
869        required = []
870        for name, val in inspect.getmembers(self.module):
871            if name == 'required':
872                required = val
873                break
874        self._required = required
875        self.deactivate_venv(dependencies=False, debug=debug)
876        return required

If the Plugin has specified dependencies in a list called required, return the list.

NOTE: Dependecies which start with 'plugin:' are Meerschaum plugins, not pip packages. Meerschaum plugins may also specify connector keys for a repo after '@'.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
  • A list of required packages and plugins (str).
def get_required_plugins(self, debug: bool = False) -> List[Plugin]: 
879    def get_required_plugins(self, debug: bool=False) -> List[mrsm.plugins.Plugin]:
880        """
881        Return a list of required Plugin objects.
882        """
883        from meerschaum.utils.warnings import warn
884        from meerschaum.config import get_config
885        from meerschaum._internal.static import STATIC_CONFIG
886        from meerschaum.connectors.parse import is_valid_connector_keys
887        plugins = []
888        _deps = self.get_dependencies(debug=debug)
889        sep = STATIC_CONFIG['plugins']['repo_separator']
890        plugin_names = [
891            _d[len('plugin:'):] for _d in _deps
892            if _d.startswith('plugin:') and len(_d) > len('plugin:')
893        ]
894        default_repo_keys = get_config('meerschaum', 'repository')
895        skipped_repo_keys = set()
896
897        for _plugin_name in plugin_names:
898            if sep in _plugin_name:
899                try:
900                    _plugin_name, _repo_keys = _plugin_name.split(sep)
901                except Exception:
902                    _repo_keys = default_repo_keys
903                    warn(
904                        f"Invalid repo keys for required plugin '{_plugin_name}'.\n    "
905                        + f"Will try to use '{_repo_keys}' instead.",
906                        stack = False,
907                    )
908            else:
909                _repo_keys = default_repo_keys
910
911            if _repo_keys in skipped_repo_keys:
912                continue
913
914            if not is_valid_connector_keys(_repo_keys):
915                warn(
916                    f"Invalid connector '{_repo_keys}'.\n"
917                    f"    Skipping required plugins from repository '{_repo_keys}'",
918                    stack=False,
919                )
920                continue
921
922            plugins.append(Plugin(_plugin_name, repo=_repo_keys))
923
924        return plugins

Return a list of required Plugin objects.

def get_required_packages(self, debug: bool = False) -> List[str]: 
927    def get_required_packages(self, debug: bool=False) -> List[str]:
928        """
929        Return the required package names (excluding plugins).
930        """
931        _deps = self.get_dependencies(debug=debug)
932        return [_d for _d in _deps if not _d.startswith('plugin:')]

Return the required package names (excluding plugins).

def activate_venv( self, dependencies: bool = True, init_if_not_exists: bool = True, debug: bool = False, **kw) -> bool: 
935    def activate_venv(
936        self,
937        dependencies: bool = True,
938        init_if_not_exists: bool = True,
939        debug: bool = False,
940        **kw
941    ) -> bool:
942        """
943        Activate the virtual environments for the plugin and its dependencies.
944
945        Parameters
946        ----------
947        dependencies: bool, default True
948            If `True`, activate the virtual environments for required plugins.
949
950        Returns
951        -------
952        A bool indicating success.
953        """
954        import meerschaum.config.paths as paths
955        from meerschaum.utils.venv import venv_target_path
956        from meerschaum.utils.packages import activate_venv
957        from meerschaum.utils.misc import make_symlink, is_symlink
958
959        if dependencies:
960            for plugin in self.get_required_plugins(debug=debug):
961                plugin.activate_venv(debug=debug, init_if_not_exists=init_if_not_exists, **kw)
962
963        vtp = venv_target_path(self.name, debug=debug, allow_nonexistent=True)
964        venv_meerschaum_path = vtp / 'meerschaum'
965
966        try:
967            success, msg = True, "Success"
968            if is_symlink(venv_meerschaum_path):
969                if pathlib.Path(os.path.realpath(venv_meerschaum_path)) != paths.PACKAGE_ROOT_PATH:
970                    venv_meerschaum_path.unlink()
971                    success, msg = make_symlink(venv_meerschaum_path, paths.PACKAGE_ROOT_PATH)
972        except Exception as e:
973            success, msg = False, str(e)
974        if not success:
975            warn(
976                f"Unable to create symlink {venv_meerschaum_path} to {paths.PACKAGE_ROOT_PATH}:\n"
977                f"{msg}"
978            )
979
980        return activate_venv(self.name, init_if_not_exists=init_if_not_exists, debug=debug, **kw)

Activate the virtual environments for the plugin and its dependencies.

Parameters
  • dependencies (bool, default True): If True, activate the virtual environments for required plugins.
Returns
  • A bool indicating success.
def deactivate_venv(self, dependencies: bool = True, debug: bool = False, **kw) -> bool: 
 983    def deactivate_venv(self, dependencies: bool=True, debug: bool = False, **kw) -> bool:
 984        """
 985        Deactivate the virtual environments for the plugin and its dependencies.
 986
 987        Parameters
 988        ----------
 989        dependencies: bool, default True
 990            If `True`, deactivate the virtual environments for required plugins.
 991
 992        Returns
 993        -------
 994        A bool indicating success.
 995        """
 996        from meerschaum.utils.packages import deactivate_venv
 997        success = deactivate_venv(self.name, debug=debug, **kw)
 998        if dependencies:
 999            for plugin in self.get_required_plugins(debug=debug):
1000                plugin.deactivate_venv(debug=debug, **kw)
1001        return success

Deactivate the virtual environments for the plugin and its dependencies.

Parameters
  • dependencies (bool, default True): If True, deactivate the virtual environments for required plugins.
Returns
  • A bool indicating success.
def install_dependencies(self, force: bool = False, debug: bool = False) -> bool: 
1004    def install_dependencies(
1005        self,
1006        force: bool = False,
1007        debug: bool = False,
1008    ) -> bool:
1009        """
1010        If specified, install dependencies.
1011        
1012        **NOTE:** Dependencies that start with `'plugin:'` will be installed as
1013        Meerschaum plugins from the same repository as this Plugin.
1014        To install from a different repository, add the repo keys after `'@'`
1015        (e.g. `'plugin:foo@api:bar'`).
1016
1017        Parameters
1018        ----------
1019        force: bool, default False
1020            If `True`, continue with the installation, even if some
1021            required packages fail to install.
1022
1023        debug: bool, default False
1024            Verbosity toggle.
1025
1026        Returns
1027        -------
1028        A bool indicating success.
1029        """
1030        from meerschaum.utils.packages import pip_install, venv_contains_package
1031        from meerschaum.utils.warnings import warn, info
1032        _deps = self.get_dependencies(debug=debug)
1033        if not _deps and self.requirements_file_path is None:
1034            return True
1035
1036        plugins = self.get_required_plugins(debug=debug)
1037        for _plugin in plugins:
1038            if _plugin.name == self.name:
1039                warn(f"Plugin '{self.name}' cannot depend on itself! Skipping...", stack=False)
1040                continue
1041            _success, _msg = _plugin.repo_connector.install_plugin(
1042                _plugin.name, debug=debug, force=force
1043            )
1044            if not _success:
1045                warn(
1046                    f"Failed to install required plugin '{_plugin}' from '{_plugin.repo_connector}'"
1047                    + f" for plugin '{self.name}':\n" + _msg,
1048                    stack = False,
1049                )
1050                if not force:
1051                    warn(
1052                        "Try installing with the `--force` flag to continue anyway.",
1053                        stack = False,
1054                    )
1055                    return False
1056                info(
1057                    "Continuing with installation despite the failure "
1058                    + "(careful, things might be broken!)...",
1059                    icon = False
1060                )
1061
1062
1063        ### First step: parse `requirements.txt` if it exists.
1064        if self.requirements_file_path is not None:
1065            if not pip_install(
1066                requirements_file_path=self.requirements_file_path,
1067                venv=self.name, debug=debug
1068            ):
1069                warn(
1070                    f"Failed to resolve 'requirements.txt' for plugin '{self.name}'.",
1071                    stack = False,
1072                )
1073                if not force:
1074                    warn(
1075                        "Try installing with `--force` to continue anyway.",
1076                        stack = False,
1077                    )
1078                    return False
1079                info(
1080                    "Continuing with installation despite the failure "
1081                    + "(careful, things might be broken!)...",
1082                    icon = False
1083                )
1084
1085
1086        ### Don't reinstall packages that are already included in required plugins.
1087        packages = []
1088        _packages = self.get_required_packages(debug=debug)
1089        accounted_for_packages = set()
1090        for package_name in _packages:
1091            for plugin in plugins:
1092                if venv_contains_package(package_name, plugin.name):
1093                    accounted_for_packages.add(package_name)
1094                    break
1095        packages = [pkg for pkg in _packages if pkg not in accounted_for_packages]
1096
1097        ### Attempt pip packages installation.
1098        if packages:
1099            for package in packages:
1100                if not pip_install(package, venv=self.name, debug=debug):
1101                    warn(
1102                        f"Failed to install required package '{package}'"
1103                        + f" for plugin '{self.name}'.",
1104                        stack = False,
1105                    )
1106                    if not force:
1107                        warn(
1108                            "Try installing with `--force` to continue anyway.",
1109                            stack = False,
1110                        )
1111                        return False
1112                    info(
1113                        "Continuing with installation despite the failure "
1114                        + "(careful, things might be broken!)...",
1115                        icon = False
1116                    )
1117        return True

If specified, install dependencies.

NOTE: Dependencies that start with 'plugin:' will be installed as Meerschaum plugins from the same repository as this Plugin. To install from a different repository, add the repo keys after '@' (e.g. 'plugin:foo@api:bar').

Parameters
  • force (bool, default False): If True, continue with the installation, even if some required packages fail to install.
  • debug (bool, default False): Verbosity toggle.
Returns
  • A bool indicating success.
full_name: str 
1120    @property
1121    def full_name(self) -> str:
1122        """
1123        Include the repo keys with the plugin's name.
1124        """
1125        from meerschaum._internal.static import STATIC_CONFIG
1126        sep = STATIC_CONFIG['plugins']['repo_separator']
1127        return self.name + sep + str(self.repo_connector)

Include the repo keys with the plugin's name.

SuccessTuple = typing.Tuple[bool, str]
class Venv: 
 19class Venv:
 20    """
 21    Manage a virtual enviroment's activation status.
 22
 23    Examples
 24    --------
 25    >>> from meerschaum.plugins import Plugin
 26    >>> with Venv('mrsm') as venv:
 27    ...     import pandas
 28    >>> with Venv(Plugin('noaa')) as venv:
 29    ...     import requests
 30    >>> venv = Venv('mrsm')
 31    >>> venv.activate()
 32    True
 33    >>> venv.deactivate()
 34    True
 35    >>> 
 36    """
 37
 38    def __init__(
 39        self,
 40        venv: Union[str, 'mrsm.core.Plugin', None] = 'mrsm',
 41        init_if_not_exists: bool = True,
 42        debug: bool = False,
 43    ) -> None:
 44        from meerschaum.utils.venv import activate_venv, deactivate_venv, active_venvs
 45        ### For some weird threading issue,
 46        ### we can't use `isinstance` here.
 47        if '_Plugin' in str(type(venv)):
 48            self._venv = venv.name
 49            self._activate = venv.activate_venv
 50            self._deactivate = venv.deactivate_venv
 51            self._kwargs = {}
 52        else:
 53            self._venv = venv
 54            self._activate = activate_venv
 55            self._deactivate = deactivate_venv
 56            self._kwargs = {'venv': venv}
 57        self._debug = debug
 58        self._init_if_not_exists = init_if_not_exists
 59        ### In case someone calls `deactivate()` before `activate()`.
 60        self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs)
 61
 62
 63    def activate(self, debug: bool = False) -> bool:
 64        """
 65        Activate this virtual environment.
 66        If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments
 67        will also be activated.
 68        """
 69        from meerschaum.utils.venv import active_venvs, init_venv
 70        self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs)
 71        try:
 72            return self._activate(
 73                debug=(debug or self._debug),
 74                init_if_not_exists=self._init_if_not_exists,
 75                **self._kwargs
 76            )
 77        except OSError as e:
 78            ### None is the current interpreter, not a named environment to recreate.
 79            if self._venv is None:
 80                raise
 81            if self._init_if_not_exists:
 82                if not init_venv(self._venv, force=True):
 83                    raise e
 84        return self._activate(
 85            debug=(debug or self._debug),
 86            init_if_not_exists=self._init_if_not_exists,
 87            **self._kwargs
 88        )
 89
 90
 91    def deactivate(self, debug: bool = False) -> bool:
 92        """
 93        Deactivate this virtual environment.
 94        If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments
 95        will also be deactivated.
 96        """
 97        return self._deactivate(debug=(debug or self._debug), **self._kwargs)
 98
 99
100    @property
101    def target_path(self) -> pathlib.Path:
102        """
103        Return the target site-packages path for this virtual environment.
104        A `meerschaum.utils.venv.Venv` may have one virtual environment per minor Python version
105        (e.g. Python 3.10 and Python 3.7).
106        """
107        from meerschaum.utils.venv import venv_target_path
108        return venv_target_path(venv=self._venv, allow_nonexistent=True, debug=self._debug)
109
110
111    @property
112    def root_path(self) -> pathlib.Path:
113        """
114        Return the top-level path for this virtual environment.
115        """
116        import meerschaum.config.paths as paths
117        if self._venv is None:
118            return self.target_path.parent
119        return paths.VIRTENV_RESOURCES_PATH / self._venv
120
121
122    def __enter__(self) -> None:
123        self.activate(debug=self._debug)
124
125
126    def __exit__(self, exc_type, exc_value, exc_traceback) -> None:
127        self.deactivate(debug=self._debug)
128
129
130    def __str__(self) -> str:
131        quote = "'" if self._venv is not None else ""
132        return "Venv(" + quote + str(self._venv) + quote + ")"
133
134
135    def __repr__(self) -> str:
136        return self.__str__()

Manage a virtual enviroment's activation status.

Examples
>>> from meerschaum.plugins import Plugin
>>> with Venv('mrsm') as venv:
...     import pandas
>>> with Venv(Plugin('noaa')) as venv:
...     import requests
>>> venv = Venv('mrsm')
>>> venv.activate()
True
>>> venv.deactivate()
True
>>>
Venv( venv: str | Plugin | None = 'mrsm', init_if_not_exists: bool = True, debug: bool = False) 
38    def __init__(
39        self,
40        venv: Union[str, 'mrsm.core.Plugin', None] = 'mrsm',
41        init_if_not_exists: bool = True,
42        debug: bool = False,
43    ) -> None:
44        from meerschaum.utils.venv import activate_venv, deactivate_venv, active_venvs
45        ### For some weird threading issue,
46        ### we can't use `isinstance` here.
47        if '_Plugin' in str(type(venv)):
48            self._venv = venv.name
49            self._activate = venv.activate_venv
50            self._deactivate = venv.deactivate_venv
51            self._kwargs = {}
52        else:
53            self._venv = venv
54            self._activate = activate_venv
55            self._deactivate = deactivate_venv
56            self._kwargs = {'venv': venv}
57        self._debug = debug
58        self._init_if_not_exists = init_if_not_exists
59        ### In case someone calls `deactivate()` before `activate()`.
60        self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs)
def activate(self, debug: bool = False) -> bool: 
63    def activate(self, debug: bool = False) -> bool:
64        """
65        Activate this virtual environment.
66        If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments
67        will also be activated.
68        """
69        from meerschaum.utils.venv import active_venvs, init_venv
70        self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs)
71        try:
72            return self._activate(
73                debug=(debug or self._debug),
74                init_if_not_exists=self._init_if_not_exists,
75                **self._kwargs
76            )
77        except OSError as e:
78            ### None is the current interpreter, not a named environment to recreate.
79            if self._venv is None:
80                raise
81            if self._init_if_not_exists:
82                if not init_venv(self._venv, force=True):
83                    raise e
84        return self._activate(
85            debug=(debug or self._debug),
86            init_if_not_exists=self._init_if_not_exists,
87            **self._kwargs
88        )

Activate this virtual environment. If a meerschaum.plugins.Plugin was provided, its dependent virtual environments will also be activated.

def deactivate(self, debug: bool = False) -> bool: 
91    def deactivate(self, debug: bool = False) -> bool:
92        """
93        Deactivate this virtual environment.
94        If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments
95        will also be deactivated.
96        """
97        return self._deactivate(debug=(debug or self._debug), **self._kwargs)

Deactivate this virtual environment. If a meerschaum.plugins.Plugin was provided, its dependent virtual environments will also be deactivated.

target_path: pathlib.Path 
100    @property
101    def target_path(self) -> pathlib.Path:
102        """
103        Return the target site-packages path for this virtual environment.
104        A `meerschaum.utils.venv.Venv` may have one virtual environment per minor Python version
105        (e.g. Python 3.10 and Python 3.7).
106        """
107        from meerschaum.utils.venv import venv_target_path
108        return venv_target_path(venv=self._venv, allow_nonexistent=True, debug=self._debug)

Return the target site-packages path for this virtual environment. A meerschaum.utils.venv.Venv may have one virtual environment per minor Python version (e.g. Python 3.10 and Python 3.7).

root_path: pathlib.Path 
111    @property
112    def root_path(self) -> pathlib.Path:
113        """
114        Return the top-level path for this virtual environment.
115        """
116        import meerschaum.config.paths as paths
117        if self._venv is None:
118            return self.target_path.parent
119        return paths.VIRTENV_RESOURCES_PATH / self._venv

Return the top-level path for this virtual environment.

class Job: 
 90class Job:
 91    """
 92    Manage a job, locally, with `systemd`, or remotely via the API.
 93    """
 94
 95    def __init__(
 96        self,
 97        name: str,
 98        sysargs: Union[List[str], str, None] = None,
 99        env: Optional[Dict[str, str]] = None,
100        executor_keys: Optional[str] = None,
101        delete_after_completion: bool = False,
102        refresh_seconds: Union[int, float, None] = None,
103        _properties: Optional[Dict[str, Any]] = None,
104        _status_hook: Optional[Callable[[], str]] = None,
105        _result_hook: Optional[Callable[[], SuccessTuple]] = None,
106        _externally_managed: bool = False,
107    ):
108        """
109        Create a new job.
110
111        Parameters
112        ----------
113        name: str
114            The name of the job to be created.
115            This will also be used as the Daemon ID.
116
117        sysargs: Union[List[str], str, None], default None
118            The sysargs of the command to be executed, e.g. 'start api'.
119
120        env: Optional[Dict[str, str]], default None
121            If provided, set these environment variables in the job's process.
122
123        executor_keys: Optional[str], default None
124            If provided, execute the job remotely on an API instance, e.g. 'api:main'
125            (or with `'systemd'`). Defaults to `'local'`, or the context's executor.
126
127        delete_after_completion: bool, default False
128            If `True`, delete this job when it has finished executing.
129
130        refresh_seconds: Union[int, float, None], default None
131            The number of seconds to sleep between refreshes.
132            Defaults to the configured value `system.cli.refresh_seconds`.
133
134        _properties: Optional[Dict[str, Any]], default None
135            If provided, use this to patch the daemon's properties.
136        """
137        native = _native.core()
138        self._core = native.Job(
139            name,
140            sysargs,
141            env=env,
142            executor_keys=executor_keys,
143            delete_after_completion=delete_after_completion,
144            refresh_seconds=refresh_seconds,
145            properties=_properties,
146            externally_managed=_externally_managed,
147            api_attributes=_native.api_attributes(executor_keys),
148        )
149        _native.emit_warnings(self._core.take_warnings())
150        self.name = name
151        self.executor_keys = self._core.executor_keys
152        self.refresh_seconds = (
153            refresh_seconds
154            if refresh_seconds is not None
155            else mrsm.get_config('system', 'cli', 'refresh_seconds')
156        )
157        if _status_hook is not None:
158            self._status_hook = _status_hook
159
160        if _result_hook is not None:
161            self._result_hook = _result_hook
162
163        self._externally_managed = _externally_managed
164
165    def _warn(self) -> None:
166        """Show the warnings the last native operation produced."""
167        _native.emit_warnings(self._core.take_warnings())
168
169    @staticmethod
170    def from_pid(pid: int, executor_keys: Optional[str] = None) -> Job:
171        """
172        Build a `Job` from the PID of a running Meerschaum process.
173
174        Parameters
175        ----------
176        pid: int
177            The PID of the process.
178
179        executor_keys: Optional[str], default None
180            The executor keys to assign to the job.
181        """
182        import os
183        try:
184            os.kill(pid, 0)
185        except ProcessLookupError as e:
186            warn(f"Process with PID {pid} does not exist.", stack=False)
187            raise e
188        except PermissionError:
189            pass
190
191        native = _native.core()
192        daemon_ids = native.jobs_daemon_ids()
193        for daemon_id in daemon_ids:
194            job = Job(daemon_id, executor_keys=executor_keys)
195            if job.pid == pid:
196                return job
197
198        cmdline_path = pathlib.Path('/proc') / str(pid) / 'cmdline'
199        if not cmdline_path.exists():
200            raise EnvironmentError(f"Is process {pid} a Meerschaum job?")
201
202        command_args = [
203            arg
204            for arg in cmdline_path.read_bytes().decode('utf-8', errors='replace').split('\0')
205            if arg
206        ]
207        # During startup the PID may change from the supervisor to its worker between
208        # the caller's PID read and the registry lookup above. Scope its persistent marker
209        # to this registry: another root may have a different Job with the same name.
210        if '--jobs-dir' in command_args:
211            import meerschaum.config.paths as paths
212            index = command_args.index('--jobs-dir') + 1
213            if (
214                index >= len(command_args)
215                or pathlib.Path(command_args[index]).resolve() != paths.DAEMON_RESOURCES_PATH.resolve()
216            ):
217                raise EnvironmentError(f"Process {pid} belongs to another Meerschaum jobs directory.")
218            for arg in command_args:
219                if arg.startswith("daemon_id='") and arg.endswith("'"):
220                    daemon_id = arg[len("daemon_id='"):-1]
221                    if daemon_id in daemon_ids:
222                        return Job(daemon_id, executor_keys=executor_keys)
223            if '__mrsm_job_supervisor__' in command_args or '__mrsm_run_daemon__' in command_args:
224                if '--daemon-id' in command_args:
225                    index = command_args.index('--daemon-id') + 1
226                    if index < len(command_args) and command_args[index] in daemon_ids:
227                        return Job(command_args[index], executor_keys=executor_keys)
228
229        if len(command_args) > 1 and command_args[1] == '-c':
230            raise EnvironmentError(f"Is process {pid} a running job?")
231
232        mrsm_ix = 0
233        for i, arg in enumerate(command_args):
234            if 'mrsm' in arg or 'meerschaum' in arg.lower():
235                mrsm_ix = i
236                break
237
238        sysargs = command_args[mrsm_ix + 1:]
239        name = None
240        for flag in ('--name', '--job-name'):
241            if flag in sysargs and sysargs.index(flag) + 1 < len(sysargs):
242                name = sysargs[sysargs.index(flag) + 1]
243                break
244        if name is None:
245            name = native.jobs_new_daemon_name()
246        return Job(name, sysargs, executor_keys=executor_keys)
247
248    def start(self, debug: bool = False) -> SuccessTuple:
249        """
250        Start the job's daemon.
251        """
252        result = self._core.start()
253        self._warn()
254        return result
255
256    def stop(
257        self,
258        timeout_seconds: Union[int, float, None] = None,
259        debug: bool = False,
260    ) -> SuccessTuple:
261        """
262        Stop the job's daemon.
263        """
264        result = self._core.stop(timeout_seconds)
265        self._warn()
266        return result
267
268    def pause(
269        self,
270        timeout_seconds: Union[int, float, None] = None,
271        debug: bool = False,
272    ) -> SuccessTuple:
273        """
274        Pause the job's daemon.
275        """
276        result = self._core.pause(timeout_seconds)
277        self._warn()
278        return result
279
280    def resume(
281        self,
282        timeout_seconds: Union[int, float, None] = None,
283        debug: bool = False,
284    ) -> SuccessTuple:
285        """
286        Resume a paused job.
287        """
288        result = self._core.resume(timeout_seconds)
289        self._warn()
290        return result
291
292    def delete(self, debug: bool = False) -> SuccessTuple:
293        """
294        Delete the job and its daemon.
295        """
296        result = self._core.delete()
297        self._warn()
298        return result
299
300    def is_running(self) -> bool:
301        """
302        Determine whether the job's daemon is running.
303        """
304        return self.status == 'running'
305
306    def exists(self, debug: bool = False) -> bool:
307        """
308        Determine whether the job exists.
309        """
310        return self._core.exists()
311
312    def get_logs(self) -> Union[str, None]:
313        """
314        Return the output text of the job's daemon.
315        """
316        return self._core.get_logs()
317
318    def monitor_logs(
319        self,
320        callback_function: Callable[[str], None] = _default_stdout_callback,
321        input_callback_function: Optional[Callable[[], str]] = None,
322        stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None,
323        stop_event: Optional[asyncio.Event] = None,
324        stop_on_exit: bool = False,
325        strip_timestamps: bool = False,
326        accept_input: bool = True,
327        debug: bool = False,
328        _wait_if_stopped: bool = True,
329    ):
330        """
331        Monitor the job's log files and execute a callback on new lines.
332
333        Parameters
334        ----------
335        callback_function: Callable[[str], None], default partial(print, end='')
336            The callback to execute as new data comes in.
337            Defaults to printing the output directly to `stdout`.
338
339        input_callback_function: Optional[Callable[[], str]], default None
340            If provided, execute this callback when the daemon is blocking on stdin.
341            Defaults to `sys.stdin.readline()`.
342
343        stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None
344            If provided, execute this callback when the daemon stops.
345            The job's SuccessTuple will be passed to the callback.
346
347        stop_event: Optional[asyncio.Event], default None
348            If provided, stop monitoring when this event is set.
349            You may instead raise `meerschaum.jobs.StopMonitoringLogs`
350            from within `callback_function` to stop monitoring.
351
352        stop_on_exit: bool, default False
353            If `True`, stop monitoring when the job stops.
354
355        strip_timestamps: bool, default False
356            If `True`, remove leading timestamps from lines.
357
358        accept_input: bool, default True
359            If `True`, accept input when the daemon blocks on stdin.
360        """
361        return self._monitor(
362            callback_function,
363            input_callback_function,
364            stop_callback_function,
365            stop_event,
366            stop_on_exit,
367            strip_timestamps,
368            accept_input,
369            _wait_if_stopped,
370            loop=None,
371            stop_flag=None,
372        )
373
374    async def monitor_logs_async(
375        self,
376        callback_function: Callable[[str], None] = _default_stdout_callback,
377        input_callback_function: Optional[Callable[[], str]] = None,
378        stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None,
379        stop_event: Optional[asyncio.Event] = None,
380        stop_on_exit: bool = False,
381        strip_timestamps: bool = False,
382        accept_input: bool = True,
383        debug: bool = False,
384        _wait_if_stopped: bool = True,
385    ):
386        """
387        Monitor the job's log files and await a callback on new lines.
388
389        Parameters
390        ----------
391        callback_function: Callable[[str], None], default _default_stdout_callback
392            The callback to execute as new data comes in.
393            Defaults to printing the output directly to `stdout`.
394
395        input_callback_function: Optional[Callable[[], str]], default None
396            If provided, execute this callback when the daemon is blocking on stdin.
397            Defaults to `sys.stdin.readline()`.
398
399        stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None
400            If provided, execute this callback when the daemon stops.
401            The job's SuccessTuple will be passed to the callback.
402
403        stop_event: Optional[asyncio.Event], default None
404            If provided, stop monitoring when this event is set.
405            You may instead raise `meerschaum.jobs.StopMonitoringLogs`
406            from within `callback_function` to stop monitoring.
407
408        stop_on_exit: bool, default False
409            If `True`, stop monitoring when the job stops.
410
411        strip_timestamps: bool, default False
412            If `True`, remove leading timestamps from lines.
413
414        accept_input: bool, default True
415            If `True`, accept input when the daemon blocks on stdin.
416        """
417        loop = asyncio.get_running_loop()
418        stop_flag = _native.core().StopFlag()
419        run = functools.partial(
420            self._monitor,
421            callback_function,
422            input_callback_function,
423            stop_callback_function,
424            stop_event,
425            stop_on_exit,
426            strip_timestamps,
427            accept_input,
428            _wait_if_stopped,
429            loop=loop,
430            stop_flag=stop_flag,
431        )
432        try:
433            await loop.run_in_executor(None, run)
434        except asyncio.CancelledError:
435            stop_flag.set()
436            raise
437        finally:
438            stop_flag.set()
439
440    def _monitor(
441        self,
442        callback_function,
443        input_callback_function,
444        stop_callback_function,
445        stop_event,
446        stop_on_exit,
447        strip_timestamps,
448        accept_input,
449        wait_if_stopped,
450        loop,
451        stop_flag,
452    ):
453        """
454        Follow the logs with the native monitor, running the callbacks on this thread's helper
455        (coroutine functions are awaited on `loop`, or on a private loop when there is none).
456        """
457        def sync(function):
458            if function is None or not inspect.iscoroutinefunction(function):
459                return function
460
461            def call(*args):
462                coroutine = function(*args)
463                if loop is not None:
464                    return asyncio.run_coroutine_threadsafe(coroutine, loop).result()
465                return asyncio.run(coroutine)
466
467            return call
468
469        def default_input_callback_function():
470            from meerschaum.utils.prompt import prompt
471            prompt_kwargs = self.get_prompt_kwargs()
472            if prompt_kwargs:
473                answer = prompt(**prompt_kwargs)
474                return answer + '\n'
475            return sys.stdin.readline()
476
477        input_callback = sync(input_callback_function) or default_input_callback_function
478        self._core.monitor_logs(
479            sync(callback_function),
480            input_callback=input_callback if accept_input else None,
481            stop_callback=sync(stop_callback_function),
482            stop_event=stop_event,
483            stop_flag=stop_flag,
484            stop_on_exit=stop_on_exit,
485            strip_timestamps=strip_timestamps,
486            accept_input=accept_input,
487            wait_if_stopped=wait_if_stopped,
488            stop_exception=StopMonitoringLogs,
489            clear_callback=_clear_screen,
490            warn=lambda message: warn(message),
491        )
492
493    def is_blocking_on_stdin(self, debug: bool = False) -> bool:
494        """
495        Return whether a job's daemon is blocking on stdin.
496        """
497        return self._core.is_blocking_on_stdin()
498
499    def get_prompt_kwargs(self, debug: bool = False) -> Dict[str, Any]:
500        """
501        Return the kwargs to the blocking `prompt()`, if available.
502        """
503        return self._core.get_prompt_kwargs()
504
505    def write_stdin(self, data):
506        """
507        Write to a job's daemon's `stdin`.
508        """
509        self._core.write_stdin(data)
510
511    @property
512    def executor(self) -> Union[Executor, None]:
513        """
514        If the job is not local, return the connector to its executor
515        (the remote API instance, `systemd`, a plugin's executor).
516        """
517        if self.executor_keys == 'local':
518            return None
519        from meerschaum.jobs._native import is_python_executor_type
520        if is_python_executor_type(self.executor_keys.split(':', maxsplit=1)[0]):
521            return mrsm.get_connector(self.executor_keys)
522        ### An executor type of a native (Rust) plugin: no Python connector exists.
523        from meerschaum.jobs._NativeExecutor import NativeExecutor
524        return NativeExecutor(self.executor_keys)
525
526    @property
527    def status(self) -> str:
528        """
529        Return the running status of the job's daemon.
530        """
531        if '_status_hook' in self.__dict__:
532            return self._status_hook()
533
534        return self._core.status
535
536    @property
537    def pid(self) -> Union[int, None]:
538        """
539        Return the PID of the job's dameon.
540        """
541        return self._core.pid
542
543    @property
544    def restart(self) -> bool:
545        """
546        Return whether to restart a stopped job.
547        """
548        return self._core.restarts
549
550    @property
551    def result(self) -> SuccessTuple:
552        """
553        Return the `SuccessTuple` when the job has terminated.
554        """
555        if '_result_hook' in self.__dict__:
556            if self.is_running():
557                return True, f"{self} is running."
558            return self._result_hook()
559
560        return self._core.result
561
562    @property
563    def sysargs(self) -> List[str]:
564        """
565        Return the sysargs to use for the Daemon.
566        """
567        return self._core.sysargs
568
569    @property
570    def health(self) -> Dict[str, Any]:
571        """
572        Return how the job's laps go (a job that repeats its work: `--loop`, `-s`, `--listen`).
573
574        Returns
575        -------
576        A dictionary with the keys:
577
578        - `state`: `'healthy'` (the last lap succeeded), `'failing'` (the last lap failed),
579          `'stalled'` (no lap for three times the expected interval, and at least a minute;
580          an hour without a sync for `--listen`; at least an hour for a first lap),
581          `'stopped'` (not running, or paused) or `'unknown'` (nothing recorded yet, e.g.
582          `'first sync in progress'`).
583        - `last_success`, `last_lap`: ISO-8601 UTC timestamps (or `None`).
584        - `consecutive_failures`: laps that failed in a row.
585        - `expected_interval_seconds`: how often laps are expected (`--min-seconds`, the
586          schedule's period, an hour for `--listen`), or `None`.
587        - `next_run`: while a running job waits for its next lap (`--min-seconds`, a cooldown
588          after a failure, a schedule's next fire), when that lap is due (ISO-8601 UTC);
589          otherwise `None`.
590        - `message`: what to show next to the state, e.g. `'last good sync 2 min ago'`.
591
592        Examples
593        --------
594        >>> job = Job('weather', 'sync pipes -c plugin:noaa --loop')
595        >>> success, msg = job.start()
596        >>> job.health['state']  # Before its first lap ends.
597        'unknown'
598        """
599        return self._core.health()
600
601    @property
602    def owner(self) -> Union[str, None]:
603        """
604        Return the user who started this job through the API server or the web console
605        (`None` for a job created locally: the CLI, Compose, systemd or Python). A job started
606        by another job has that job's owner. For an `api:` job, the server's record.
607
608        Examples
609        --------
610        >>> Job('nightly', executor_keys='api:main').owner
611        'alice'
612        """
613        return self._core.access().get('owner', None)
614
615    @property
616    def team(self) -> Union[str, None]:
617        """
618        Return the team this job belongs to: the primary team of the user who started it through
619        the API server or the web console (a job started by another job: that job's team).
620        `None` means Everyone (every local job).
621
622        Members of the team (and administrators) see the job and, with the job permissions,
623        control it. Renaming the team renames it here; deleting the team moves the job to
624        Everyone.
625        """
626        return self._core.access().get('team', None)
627
628    def get_daemon_properties(self) -> Dict[str, Any]:
629        """
630        Return the `properties` dictionary for the job's daemon.
631        """
632        return self._core.get_daemon_properties()
633
634    @property
635    def daemon(self) -> 'Daemon':
636        """
637        Return the daemon which this job manages.
638        """
639        from meerschaum.utils.daemon import Daemon
640        if self.executor_keys == 'local' and self.exists():
641            return Daemon(daemon_id=self.name)
642
643        return Daemon(
644            daemon_id=self.name,
645            sysargs=self.sysargs,
646            label=shlex.join(self.sysargs),
647            properties=self.get_daemon_properties(),
648        )
649
650    @property
651    def began(self) -> Union[datetime, None]:
652        """
653        The datetime when the job began running.
654        """
655        return _parse_dt(self._core.began)
656
657    @property
658    def ended(self) -> Union[datetime, None]:
659        """
660        The datetime when the job stopped running.
661        """
662        return _parse_dt(self._core.ended)
663
664    @property
665    def paused(self) -> Union[datetime, None]:
666        """
667        The datetime when the job was suspended while running.
668        """
669        return _parse_dt(self._core.paused)
670
671    @property
672    def stop_time(self) -> Union[datetime, None]:
673        """
674        Return the timestamp when the job was manually stopped.
675        """
676        return _parse_dt(self._core.stop_time)
677
678    @property
679    def hidden(self) -> bool:
680        """
681        Return a bool indicating whether this job should be displayed.
682        """
683        return self._core.hidden
684
685    def check_restart(self) -> SuccessTuple:
686        """
687        If `restart` is `True` and the daemon is not running,
688        restart the job.
689        Do not restart if the job was manually stopped.
690        """
691        result = self._core.check_restart()
692        self._warn()
693        return result
694
695    @property
696    def label(self) -> str:
697        """
698        Return the job's Daemon label (joined sysargs).
699        """
700        return self._core.label
701
702    @property
703    def _externally_managed_file(self) -> pathlib.Path:
704        """
705        Return the path to the externally managed file.
706        """
707        return pathlib.Path(self._core.paths()['externally_managed'])
708
709    def _set_externally_managed(self):
710        """
711        Set this job as externally managed.
712        """
713        self._externally_managed = True
714        self._core.set_externally_managed()
715
716    @property
717    def _is_externally_managed(self) -> bool:
718        """
719        Return whether this job is externally managed.
720        """
721        return self._core.is_externally_managed
722
723    @property
724    def env(self) -> Dict[str, str]:
725        """
726        Return the environment variables to set for the job's process.
727        """
728        return self._core.env
729
730    @property
731    def delete_after_completion(self) -> bool:
732        """
733        Return whether this job is configured to delete itself after completion.
734        """
735        return self._core.delete_after_completion
736
737    def __str__(self) -> str:
738        return self._core.describe()
739
740    def __repr__(self) -> str:
741        return str(self)
742
743    def __hash__(self) -> int:
744        return hash(self.name)

Manage a job, locally, with systemd, or remotely via the API.

Job( name: str, sysargs: List[str] | str | None = None, env: Dict[str, str] | None = None, executor_keys: str | None = None, delete_after_completion: bool = False, refresh_seconds: int | float | None = None, _properties: Dict[str, Any] | None = None, _status_hook: Callable[[], str] | None = None, _result_hook: Callable[[], Tuple[bool, str]] | None = None, _externally_managed: bool = False) 
 95    def __init__(
 96        self,
 97        name: str,
 98        sysargs: Union[List[str], str, None] = None,
 99        env: Optional[Dict[str, str]] = None,
100        executor_keys: Optional[str] = None,
101        delete_after_completion: bool = False,
102        refresh_seconds: Union[int, float, None] = None,
103        _properties: Optional[Dict[str, Any]] = None,
104        _status_hook: Optional[Callable[[], str]] = None,
105        _result_hook: Optional[Callable[[], SuccessTuple]] = None,
106        _externally_managed: bool = False,
107    ):
108        """
109        Create a new job.
110
111        Parameters
112        ----------
113        name: str
114            The name of the job to be created.
115            This will also be used as the Daemon ID.
116
117        sysargs: Union[List[str], str, None], default None
118            The sysargs of the command to be executed, e.g. 'start api'.
119
120        env: Optional[Dict[str, str]], default None
121            If provided, set these environment variables in the job's process.
122
123        executor_keys: Optional[str], default None
124            If provided, execute the job remotely on an API instance, e.g. 'api:main'
125            (or with `'systemd'`). Defaults to `'local'`, or the context's executor.
126
127        delete_after_completion: bool, default False
128            If `True`, delete this job when it has finished executing.
129
130        refresh_seconds: Union[int, float, None], default None
131            The number of seconds to sleep between refreshes.
132            Defaults to the configured value `system.cli.refresh_seconds`.
133
134        _properties: Optional[Dict[str, Any]], default None
135            If provided, use this to patch the daemon's properties.
136        """
137        native = _native.core()
138        self._core = native.Job(
139            name,
140            sysargs,
141            env=env,
142            executor_keys=executor_keys,
143            delete_after_completion=delete_after_completion,
144            refresh_seconds=refresh_seconds,
145            properties=_properties,
146            externally_managed=_externally_managed,
147            api_attributes=_native.api_attributes(executor_keys),
148        )
149        _native.emit_warnings(self._core.take_warnings())
150        self.name = name
151        self.executor_keys = self._core.executor_keys
152        self.refresh_seconds = (
153            refresh_seconds
154            if refresh_seconds is not None
155            else mrsm.get_config('system', 'cli', 'refresh_seconds')
156        )
157        if _status_hook is not None:
158            self._status_hook = _status_hook
159
160        if _result_hook is not None:
161            self._result_hook = _result_hook
162
163        self._externally_managed = _externally_managed

Create a new job.

Parameters
  • name (str): The name of the job to be created. This will also be used as the Daemon ID.
  • sysargs (Union[List[str], str, None], default None): The sysargs of the command to be executed, e.g. 'start api'.
  • env (Optional[Dict[str, str]], default None): If provided, set these environment variables in the job's process.
  • executor_keys (Optional[str], default None): If provided, execute the job remotely on an API instance, e.g. 'api:main' (or with 'systemd'). Defaults to 'local', or the context's executor.
  • delete_after_completion (bool, default False): If True, delete this job when it has finished executing.
  • refresh_seconds (Union[int, float, None], default None): The number of seconds to sleep between refreshes. Defaults to the configured value system.cli.refresh_seconds.
  • _properties (Optional[Dict[str, Any]], default None): If provided, use this to patch the daemon's properties.
name
executor_keys
refresh_seconds
@staticmethod
def from_pid(pid: int, executor_keys: str | None = None) -> Job: 
169    @staticmethod
170    def from_pid(pid: int, executor_keys: Optional[str] = None) -> Job:
171        """
172        Build a `Job` from the PID of a running Meerschaum process.
173
174        Parameters
175        ----------
176        pid: int
177            The PID of the process.
178
179        executor_keys: Optional[str], default None
180            The executor keys to assign to the job.
181        """
182        import os
183        try:
184            os.kill(pid, 0)
185        except ProcessLookupError as e:
186            warn(f"Process with PID {pid} does not exist.", stack=False)
187            raise e
188        except PermissionError:
189            pass
190
191        native = _native.core()
192        daemon_ids = native.jobs_daemon_ids()
193        for daemon_id in daemon_ids:
194            job = Job(daemon_id, executor_keys=executor_keys)
195            if job.pid == pid:
196                return job
197
198        cmdline_path = pathlib.Path('/proc') / str(pid) / 'cmdline'
199        if not cmdline_path.exists():
200            raise EnvironmentError(f"Is process {pid} a Meerschaum job?")
201
202        command_args = [
203            arg
204            for arg in cmdline_path.read_bytes().decode('utf-8', errors='replace').split('\0')
205            if arg
206        ]
207        # During startup the PID may change from the supervisor to its worker between
208        # the caller's PID read and the registry lookup above. Scope its persistent marker
209        # to this registry: another root may have a different Job with the same name.
210        if '--jobs-dir' in command_args:
211            import meerschaum.config.paths as paths
212            index = command_args.index('--jobs-dir') + 1
213            if (
214                index >= len(command_args)
215                or pathlib.Path(command_args[index]).resolve() != paths.DAEMON_RESOURCES_PATH.resolve()
216            ):
217                raise EnvironmentError(f"Process {pid} belongs to another Meerschaum jobs directory.")
218            for arg in command_args:
219                if arg.startswith("daemon_id='") and arg.endswith("'"):
220                    daemon_id = arg[len("daemon_id='"):-1]
221                    if daemon_id in daemon_ids:
222                        return Job(daemon_id, executor_keys=executor_keys)
223            if '__mrsm_job_supervisor__' in command_args or '__mrsm_run_daemon__' in command_args:
224                if '--daemon-id' in command_args:
225                    index = command_args.index('--daemon-id') + 1
226                    if index < len(command_args) and command_args[index] in daemon_ids:
227                        return Job(command_args[index], executor_keys=executor_keys)
228
229        if len(command_args) > 1 and command_args[1] == '-c':
230            raise EnvironmentError(f"Is process {pid} a running job?")
231
232        mrsm_ix = 0
233        for i, arg in enumerate(command_args):
234            if 'mrsm' in arg or 'meerschaum' in arg.lower():
235                mrsm_ix = i
236                break
237
238        sysargs = command_args[mrsm_ix + 1:]
239        name = None
240        for flag in ('--name', '--job-name'):
241            if flag in sysargs and sysargs.index(flag) + 1 < len(sysargs):
242                name = sysargs[sysargs.index(flag) + 1]
243                break
244        if name is None:
245            name = native.jobs_new_daemon_name()
246        return Job(name, sysargs, executor_keys=executor_keys)

Build a Job from the PID of a running Meerschaum process.

Parameters
  • pid (int): The PID of the process.
  • executor_keys (Optional[str], default None): The executor keys to assign to the job.
def start(self, debug: bool = False) -> Tuple[bool, str]: 
248    def start(self, debug: bool = False) -> SuccessTuple:
249        """
250        Start the job's daemon.
251        """
252        result = self._core.start()
253        self._warn()
254        return result

Start the job's daemon.

def stop( self, timeout_seconds: int | float | None = None, debug: bool = False) -> Tuple[bool, str]: 
256    def stop(
257        self,
258        timeout_seconds: Union[int, float, None] = None,
259        debug: bool = False,
260    ) -> SuccessTuple:
261        """
262        Stop the job's daemon.
263        """
264        result = self._core.stop(timeout_seconds)
265        self._warn()
266        return result

Stop the job's daemon.

def pause( self, timeout_seconds: int | float | None = None, debug: bool = False) -> Tuple[bool, str]: 
268    def pause(
269        self,
270        timeout_seconds: Union[int, float, None] = None,
271        debug: bool = False,
272    ) -> SuccessTuple:
273        """
274        Pause the job's daemon.
275        """
276        result = self._core.pause(timeout_seconds)
277        self._warn()
278        return result

Pause the job's daemon.

def resume( self, timeout_seconds: int | float | None = None, debug: bool = False) -> Tuple[bool, str]: 
280    def resume(
281        self,
282        timeout_seconds: Union[int, float, None] = None,
283        debug: bool = False,
284    ) -> SuccessTuple:
285        """
286        Resume a paused job.
287        """
288        result = self._core.resume(timeout_seconds)
289        self._warn()
290        return result

Resume a paused job.

def delete(self, debug: bool = False) -> Tuple[bool, str]: 
292    def delete(self, debug: bool = False) -> SuccessTuple:
293        """
294        Delete the job and its daemon.
295        """
296        result = self._core.delete()
297        self._warn()
298        return result

Delete the job and its daemon.

def is_running(self) -> bool: 
300    def is_running(self) -> bool:
301        """
302        Determine whether the job's daemon is running.
303        """
304        return self.status == 'running'

Determine whether the job's daemon is running.

def exists(self, debug: bool = False) -> bool: 
306    def exists(self, debug: bool = False) -> bool:
307        """
308        Determine whether the job exists.
309        """
310        return self._core.exists()

Determine whether the job exists.

def get_logs(self) -> str | None: 
312    def get_logs(self) -> Union[str, None]:
313        """
314        Return the output text of the job's daemon.
315        """
316        return self._core.get_logs()

Return the output text of the job's daemon.

def monitor_logs( self, callback_function: Callable[[str], NoneType] = <function _default_stdout_callback>, input_callback_function: Callable[[], str] | None = None, stop_callback_function: Callable[[Tuple[bool, str]], NoneType] | None = None, stop_event: asyncio.locks.Event | None = None, stop_on_exit: bool = False, strip_timestamps: bool = False, accept_input: bool = True, debug: bool = False, _wait_if_stopped: bool = True): 
318    def monitor_logs(
319        self,
320        callback_function: Callable[[str], None] = _default_stdout_callback,
321        input_callback_function: Optional[Callable[[], str]] = None,
322        stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None,
323        stop_event: Optional[asyncio.Event] = None,
324        stop_on_exit: bool = False,
325        strip_timestamps: bool = False,
326        accept_input: bool = True,
327        debug: bool = False,
328        _wait_if_stopped: bool = True,
329    ):
330        """
331        Monitor the job's log files and execute a callback on new lines.
332
333        Parameters
334        ----------
335        callback_function: Callable[[str], None], default partial(print, end='')
336            The callback to execute as new data comes in.
337            Defaults to printing the output directly to `stdout`.
338
339        input_callback_function: Optional[Callable[[], str]], default None
340            If provided, execute this callback when the daemon is blocking on stdin.
341            Defaults to `sys.stdin.readline()`.
342
343        stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None
344            If provided, execute this callback when the daemon stops.
345            The job's SuccessTuple will be passed to the callback.
346
347        stop_event: Optional[asyncio.Event], default None
348            If provided, stop monitoring when this event is set.
349            You may instead raise `meerschaum.jobs.StopMonitoringLogs`
350            from within `callback_function` to stop monitoring.
351
352        stop_on_exit: bool, default False
353            If `True`, stop monitoring when the job stops.
354
355        strip_timestamps: bool, default False
356            If `True`, remove leading timestamps from lines.
357
358        accept_input: bool, default True
359            If `True`, accept input when the daemon blocks on stdin.
360        """
361        return self._monitor(
362            callback_function,
363            input_callback_function,
364            stop_callback_function,
365            stop_event,
366            stop_on_exit,
367            strip_timestamps,
368            accept_input,
369            _wait_if_stopped,
370            loop=None,
371            stop_flag=None,
372        )

Monitor the job's log files and execute a callback on new lines.

Parameters
  • callback_function (Callable[[str], None], default partial(print, end='')): The callback to execute as new data comes in. Defaults to printing the output directly to stdout.
  • input_callback_function (Optional[Callable[[], str]], default None): If provided, execute this callback when the daemon is blocking on stdin. Defaults to sys.stdin.readline().
  • stop_callback_function (Optional[Callable[[SuccessTuple]], str], default None): If provided, execute this callback when the daemon stops. The job's SuccessTuple will be passed to the callback.
  • stop_event (Optional[asyncio.Event], default None): If provided, stop monitoring when this event is set. You may instead raise meerschaum.jobs.StopMonitoringLogs from within callback_function to stop monitoring.
  • stop_on_exit (bool, default False): If True, stop monitoring when the job stops.
  • strip_timestamps (bool, default False): If True, remove leading timestamps from lines.
  • accept_input (bool, default True): If True, accept input when the daemon blocks on stdin.
async def monitor_logs_async( self, callback_function: Callable[[str], NoneType] = <function _default_stdout_callback>, input_callback_function: Callable[[], str] | None = None, stop_callback_function: Callable[[Tuple[bool, str]], NoneType] | None = None, stop_event: asyncio.locks.Event | None = None, stop_on_exit: bool = False, strip_timestamps: bool = False, accept_input: bool = True, debug: bool = False, _wait_if_stopped: bool = True): 
374    async def monitor_logs_async(
375        self,
376        callback_function: Callable[[str], None] = _default_stdout_callback,
377        input_callback_function: Optional[Callable[[], str]] = None,
378        stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None,
379        stop_event: Optional[asyncio.Event] = None,
380        stop_on_exit: bool = False,
381        strip_timestamps: bool = False,
382        accept_input: bool = True,
383        debug: bool = False,
384        _wait_if_stopped: bool = True,
385    ):
386        """
387        Monitor the job's log files and await a callback on new lines.
388
389        Parameters
390        ----------
391        callback_function: Callable[[str], None], default _default_stdout_callback
392            The callback to execute as new data comes in.
393            Defaults to printing the output directly to `stdout`.
394
395        input_callback_function: Optional[Callable[[], str]], default None
396            If provided, execute this callback when the daemon is blocking on stdin.
397            Defaults to `sys.stdin.readline()`.
398
399        stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None
400            If provided, execute this callback when the daemon stops.
401            The job's SuccessTuple will be passed to the callback.
402
403        stop_event: Optional[asyncio.Event], default None
404            If provided, stop monitoring when this event is set.
405            You may instead raise `meerschaum.jobs.StopMonitoringLogs`
406            from within `callback_function` to stop monitoring.
407
408        stop_on_exit: bool, default False
409            If `True`, stop monitoring when the job stops.
410
411        strip_timestamps: bool, default False
412            If `True`, remove leading timestamps from lines.
413
414        accept_input: bool, default True
415            If `True`, accept input when the daemon blocks on stdin.
416        """
417        loop = asyncio.get_running_loop()
418        stop_flag = _native.core().StopFlag()
419        run = functools.partial(
420            self._monitor,
421            callback_function,
422            input_callback_function,
423            stop_callback_function,
424            stop_event,
425            stop_on_exit,
426            strip_timestamps,
427            accept_input,
428            _wait_if_stopped,
429            loop=loop,
430            stop_flag=stop_flag,
431        )
432        try:
433            await loop.run_in_executor(None, run)
434        except asyncio.CancelledError:
435            stop_flag.set()
436            raise
437        finally:
438            stop_flag.set()

Monitor the job's log files and await a callback on new lines.

Parameters
  • callback_function (Callable[[str], None], default _default_stdout_callback): The callback to execute as new data comes in. Defaults to printing the output directly to stdout.
  • input_callback_function (Optional[Callable[[], str]], default None): If provided, execute this callback when the daemon is blocking on stdin. Defaults to sys.stdin.readline().
  • stop_callback_function (Optional[Callable[[SuccessTuple]], str], default None): If provided, execute this callback when the daemon stops. The job's SuccessTuple will be passed to the callback.
  • stop_event (Optional[asyncio.Event], default None): If provided, stop monitoring when this event is set. You may instead raise meerschaum.jobs.StopMonitoringLogs from within callback_function to stop monitoring.
  • stop_on_exit (bool, default False): If True, stop monitoring when the job stops.
  • strip_timestamps (bool, default False): If True, remove leading timestamps from lines.
  • accept_input (bool, default True): If True, accept input when the daemon blocks on stdin.
def is_blocking_on_stdin(self, debug: bool = False) -> bool: 
493    def is_blocking_on_stdin(self, debug: bool = False) -> bool:
494        """
495        Return whether a job's daemon is blocking on stdin.
496        """
497        return self._core.is_blocking_on_stdin()

Return whether a job's daemon is blocking on stdin.

def get_prompt_kwargs(self, debug: bool = False) -> Dict[str, Any]: 
499    def get_prompt_kwargs(self, debug: bool = False) -> Dict[str, Any]:
500        """
501        Return the kwargs to the blocking `prompt()`, if available.
502        """
503        return self._core.get_prompt_kwargs()

Return the kwargs to the blocking prompt(), if available.

def write_stdin(self, data): 
505    def write_stdin(self, data):
506        """
507        Write to a job's daemon's `stdin`.
508        """
509        self._core.write_stdin(data)

Write to a job's daemon's stdin.

executor: meerschaum.jobs.Executor | None 
511    @property
512    def executor(self) -> Union[Executor, None]:
513        """
514        If the job is not local, return the connector to its executor
515        (the remote API instance, `systemd`, a plugin's executor).
516        """
517        if self.executor_keys == 'local':
518            return None
519        from meerschaum.jobs._native import is_python_executor_type
520        if is_python_executor_type(self.executor_keys.split(':', maxsplit=1)[0]):
521            return mrsm.get_connector(self.executor_keys)
522        ### An executor type of a native (Rust) plugin: no Python connector exists.
523        from meerschaum.jobs._NativeExecutor import NativeExecutor
524        return NativeExecutor(self.executor_keys)

If the job is not local, return the connector to its executor (the remote API instance, systemd, a plugin's executor).

status: str 
526    @property
527    def status(self) -> str:
528        """
529        Return the running status of the job's daemon.
530        """
531        if '_status_hook' in self.__dict__:
532            return self._status_hook()
533
534        return self._core.status

Return the running status of the job's daemon.

pid: int | None 
536    @property
537    def pid(self) -> Union[int, None]:
538        """
539        Return the PID of the job's dameon.
540        """
541        return self._core.pid

Return the PID of the job's dameon.

restart: bool 
543    @property
544    def restart(self) -> bool:
545        """
546        Return whether to restart a stopped job.
547        """
548        return self._core.restarts

Return whether to restart a stopped job.

result: Tuple[bool, str] 
550    @property
551    def result(self) -> SuccessTuple:
552        """
553        Return the `SuccessTuple` when the job has terminated.
554        """
555        if '_result_hook' in self.__dict__:
556            if self.is_running():
557                return True, f"{self} is running."
558            return self._result_hook()
559
560        return self._core.result

Return the SuccessTuple when the job has terminated.

sysargs: List[str] 
562    @property
563    def sysargs(self) -> List[str]:
564        """
565        Return the sysargs to use for the Daemon.
566        """
567        return self._core.sysargs

Return the sysargs to use for the Daemon.

health: Dict[str, Any] 
569    @property
570    def health(self) -> Dict[str, Any]:
571        """
572        Return how the job's laps go (a job that repeats its work: `--loop`, `-s`, `--listen`).
573
574        Returns
575        -------
576        A dictionary with the keys:
577
578        - `state`: `'healthy'` (the last lap succeeded), `'failing'` (the last lap failed),
579          `'stalled'` (no lap for three times the expected interval, and at least a minute;
580          an hour without a sync for `--listen`; at least an hour for a first lap),
581          `'stopped'` (not running, or paused) or `'unknown'` (nothing recorded yet, e.g.
582          `'first sync in progress'`).
583        - `last_success`, `last_lap`: ISO-8601 UTC timestamps (or `None`).
584        - `consecutive_failures`: laps that failed in a row.
585        - `expected_interval_seconds`: how often laps are expected (`--min-seconds`, the
586          schedule's period, an hour for `--listen`), or `None`.
587        - `next_run`: while a running job waits for its next lap (`--min-seconds`, a cooldown
588          after a failure, a schedule's next fire), when that lap is due (ISO-8601 UTC);
589          otherwise `None`.
590        - `message`: what to show next to the state, e.g. `'last good sync 2 min ago'`.
591
592        Examples
593        --------
594        >>> job = Job('weather', 'sync pipes -c plugin:noaa --loop')
595        >>> success, msg = job.start()
596        >>> job.health['state']  # Before its first lap ends.
597        'unknown'
598        """
599        return self._core.health()

Return how the job's laps go (a job that repeats its work: --loop, -s, --listen).

Returns
  • A dictionary with the keys:
  • - state ('healthy' (the last lap succeeded), 'failing' (the last lap failed),): 'stalled' (no lap for three times the expected interval, and at least a minute; an hour without a sync for --listen; at least an hour for a first lap), 'stopped' (not running, or paused) or 'unknown' (nothing recorded yet, e.g. 'first sync in progress').

  • - last_success, last_lap (ISO-8601 UTC timestamps (or None).):

  • - consecutive_failures (laps that failed in a row.):

  • - expected_interval_seconds (how often laps are expected (--min-seconds, the): schedule's period, an hour for --listen), or None.

  • - next_run (while a running job waits for its next lap (--min-seconds, a cooldown): after a failure, a schedule's next fire), when that lap is due (ISO-8601 UTC); otherwise None.
  • - message (what to show next to the state, e.g. 'last good sync 2 min ago'.):
Examples
>>> job = Job('weather', 'sync pipes -c plugin:noaa --loop')
>>> success, msg = job.start()
>>> job.health['state']  # Before its first lap ends.
'unknown'
owner: str | None 
601    @property
602    def owner(self) -> Union[str, None]:
603        """
604        Return the user who started this job through the API server or the web console
605        (`None` for a job created locally: the CLI, Compose, systemd or Python). A job started
606        by another job has that job's owner. For an `api:` job, the server's record.
607
608        Examples
609        --------
610        >>> Job('nightly', executor_keys='api:main').owner
611        'alice'
612        """
613        return self._core.access().get('owner', None)

Return the user who started this job through the API server or the web console (None for a job created locally: the CLI, Compose, systemd or Python). A job started by another job has that job's owner. For an api: job, the server's record.

Examples
>>> Job('nightly', executor_keys='api:main').owner
'alice'
team: str | None 
615    @property
616    def team(self) -> Union[str, None]:
617        """
618        Return the team this job belongs to: the primary team of the user who started it through
619        the API server or the web console (a job started by another job: that job's team).
620        `None` means Everyone (every local job).
621
622        Members of the team (and administrators) see the job and, with the job permissions,
623        control it. Renaming the team renames it here; deleting the team moves the job to
624        Everyone.
625        """
626        return self._core.access().get('team', None)

Return the team this job belongs to: the primary team of the user who started it through the API server or the web console (a job started by another job: that job's team). None means Everyone (every local job).

Members of the team (and administrators) see the job and, with the job permissions, control it. Renaming the team renames it here; deleting the team moves the job to Everyone.

def get_daemon_properties(self) -> Dict[str, Any]: 
628    def get_daemon_properties(self) -> Dict[str, Any]:
629        """
630        Return the `properties` dictionary for the job's daemon.
631        """
632        return self._core.get_daemon_properties()

Return the properties dictionary for the job's daemon.

daemon: meerschaum.utils.daemon.Daemon 
634    @property
635    def daemon(self) -> 'Daemon':
636        """
637        Return the daemon which this job manages.
638        """
639        from meerschaum.utils.daemon import Daemon
640        if self.executor_keys == 'local' and self.exists():
641            return Daemon(daemon_id=self.name)
642
643        return Daemon(
644            daemon_id=self.name,
645            sysargs=self.sysargs,
646            label=shlex.join(self.sysargs),
647            properties=self.get_daemon_properties(),
648        )

Return the daemon which this job manages.

began: datetime.datetime | None 
650    @property
651    def began(self) -> Union[datetime, None]:
652        """
653        The datetime when the job began running.
654        """
655        return _parse_dt(self._core.began)

The datetime when the job began running.

ended: datetime.datetime | None 
657    @property
658    def ended(self) -> Union[datetime, None]:
659        """
660        The datetime when the job stopped running.
661        """
662        return _parse_dt(self._core.ended)

The datetime when the job stopped running.

paused: datetime.datetime | None 
664    @property
665    def paused(self) -> Union[datetime, None]:
666        """
667        The datetime when the job was suspended while running.
668        """
669        return _parse_dt(self._core.paused)

The datetime when the job was suspended while running.

stop_time: datetime.datetime | None 
671    @property
672    def stop_time(self) -> Union[datetime, None]:
673        """
674        Return the timestamp when the job was manually stopped.
675        """
676        return _parse_dt(self._core.stop_time)

Return the timestamp when the job was manually stopped.

hidden: bool 
678    @property
679    def hidden(self) -> bool:
680        """
681        Return a bool indicating whether this job should be displayed.
682        """
683        return self._core.hidden

Return a bool indicating whether this job should be displayed.

def check_restart(self) -> Tuple[bool, str]: 
685    def check_restart(self) -> SuccessTuple:
686        """
687        If `restart` is `True` and the daemon is not running,
688        restart the job.
689        Do not restart if the job was manually stopped.
690        """
691        result = self._core.check_restart()
692        self._warn()
693        return result

If restart is True and the daemon is not running, restart the job. Do not restart if the job was manually stopped.

label: str 
695    @property
696    def label(self) -> str:
697        """
698        Return the job's Daemon label (joined sysargs).
699        """
700        return self._core.label

Return the job's Daemon label (joined sysargs).

env: Dict[str, str] 
723    @property
724    def env(self) -> Dict[str, str]:
725        """
726        Return the environment variables to set for the job's process.
727        """
728        return self._core.env

Return the environment variables to set for the job's process.

delete_after_completion: bool 
730    @property
731    def delete_after_completion(self) -> bool:
732        """
733        Return whether this job is configured to delete itself after completion.
734        """
735        return self._core.delete_after_completion

Return whether this job is configured to delete itself after completion.

def make_executor(cls): 
296def make_executor(cls):
297    """
298    Register a class as an `Executor`.
299    """
300    import re
301    from meerschaum.connectors import make_connector
302    suffix_regex = r'executor$'
303    typ = re.sub(suffix_regex, '', cls.__name__.lower())
304    if typ not in executor_types:
305        executor_types.append(typ)
306    return make_connector(cls, _is_executor=True)

Register a class as an Executor.

def pprint( *args, detect_password: bool = True, nopretty: bool = False, **kw) -> None: 
 10def pprint(
 11    *args,
 12    detect_password: bool = True,
 13    nopretty: bool = False,
 14    **kw
 15) -> None:
 16    """Pretty print an object according to the configured ANSI and UNICODE settings.
 17    If detect_password is True (default), search and replace passwords with '*' characters.
 18    Does not mutate objects.
 19    """
 20    import copy
 21    import json
 22    from meerschaum.utils.packages import attempt_import, import_rich
 23    from meerschaum.utils.formatting import ANSI, get_console, print_tuple
 24    from meerschaum.utils.warnings import error
 25    from meerschaum.utils.misc import replace_password, dict_from_od, filter_keywords
 26    from collections import OrderedDict
 27
 28    if (
 29        len(args) == 1
 30        and
 31        isinstance(args[0], tuple)
 32        and
 33        len(args[0]) == 2
 34        and
 35        isinstance(args[0][0], bool)
 36        and
 37        isinstance(args[0][1], str)
 38    ):
 39        return print_tuple(args[0], **filter_keywords(print_tuple, **kw))
 40
 41    modify = True
 42    rich_pprint = None
 43    if ANSI and not nopretty:
 44        ### `rich` is not a dependency of the base wheel: without it, output is not styled.
 45        rich = import_rich(lazy=False)
 46        rich_pretty = attempt_import('rich.pretty', lazy=False) if rich is not None else None
 47        if rich_pretty is not None and get_console() is not None:
 48            def _rich_pprint(*args, **kw):
 49                _console = get_console()
 50                _kw = filter_keywords(_console.print, **kw)
 51                _console.print(*args, **_kw)
 52            rich_pprint = _rich_pprint
 53    if rich_pprint is None and not nopretty:
 54        pprintpp = attempt_import('pprintpp', warn=False)
 55        try:
 56            _pprint = pprintpp.pprint
 57        except Exception :
 58            import pprint as _pprint_module
 59            _pprint = _pprint_module.pprint
 60
 61    func = (
 62        _pprint if rich_pprint is None else rich_pprint
 63    ) if not nopretty else print
 64
 65    try:
 66        args_copy = copy.deepcopy(args)
 67    except Exception:
 68        args_copy = args
 69        modify = False
 70
 71    _args = []
 72    for a in args:
 73        c = a
 74        ### convert OrderedDict into dict
 75        if isinstance(a, OrderedDict) or issubclass(type(a), OrderedDict):
 76            c = dict_from_od(copy.deepcopy(c))
 77        _args.append(c)
 78    args = _args
 79
 80    _args = list(args)
 81    if detect_password and modify:
 82        _args = []
 83        for a in args:
 84            c = a
 85            if isinstance(c, dict):
 86                c = replace_password(copy.deepcopy(c))
 87            if nopretty:
 88                try:
 89                    c = json.dumps(c)
 90                    is_json = True
 91                except Exception:
 92                    is_json = False
 93                if not is_json:
 94                    try:
 95                        c = str(c)
 96                    except Exception:
 97                        pass
 98            _args.append(c)
 99
100    ### filter out unsupported keywords
101    func_kw = filter_keywords(func, **kw) if not nopretty else {}
102    error_msg = None
103    try:
104        func(*_args, **func_kw)
105    except Exception as e:
106        error_msg = e
107    if error_msg is not None:
108        error(error_msg)

Pretty print an object according to the configured ANSI and UNICODE settings. If detect_password is True (default), search and replace passwords with '*' characters. Does not mutate objects.

def attempt_import( *names: str, lazy: bool = True, warn: bool = True, install: bool = True, venv: str | None = 'mrsm', precheck: bool = True, split: bool = True, check_update: bool = False, check_pypi: bool = False, check_is_installed: bool = True, allow_outside_venv: bool = True, color: bool = True, debug: bool = False) -> Any: 
1458def attempt_import(
1459    *names: str,
1460    lazy: bool = True,
1461    warn: bool = True,
1462    install: bool = True,
1463    venv: Optional[str] = 'mrsm',
1464    precheck: bool = True,
1465    split: bool = True,
1466    check_update: bool = False,
1467    check_pypi: bool = False,
1468    check_is_installed: bool = True,
1469    allow_outside_venv: bool = True,
1470    color: bool = True,
1471    debug: bool = False
1472) -> Any:
1473    """
1474    Raise a warning if packages are not installed; otherwise import and return modules.
1475    If `lazy` is `True`, return lazy-imported modules.
1476    
1477    Returns tuple of modules if multiple names are provided, else returns one module.
1478    
1479    Parameters
1480    ----------
1481    names: List[str]
1482        The packages to be imported.
1483
1484    lazy: bool, default True
1485        If `True`, lazily load packages.
1486
1487    warn: bool, default True
1488        If `True`, raise a warning if a package cannot be imported.
1489
1490    install: bool, default True
1491        If `True`, attempt to install a missing package into the designated virtual environment.
1492        If `check_update` is True, install updates if available.
1493
1494    venv: Optional[str], default 'mrsm'
1495        The virtual environment in which to search for packages and to install packages into.
1496
1497    precheck: bool, default True
1498        If `True`, attempt to find module before importing (necessary for checking if modules exist
1499        and retaining lazy imports), otherwise assume lazy is `False`.
1500
1501    split: bool, default True
1502        If `True`, split packages' names on `'.'`.
1503
1504    check_update: bool, default False
1505        If `True` and `install` is `True`, install updates if the required minimum version
1506        does not match.
1507
1508    check_pypi: bool, default False
1509        If `True` and `check_update` is `True`, check PyPI when determining whether
1510        an update is required.
1511
1512    check_is_installed: bool, default True
1513        If `True`, check if the package is contained in the virtual environment.
1514
1515    allow_outside_venv: bool, default True
1516        If `True`, search outside of the specified virtual environment
1517        if the package cannot be found.
1518        Setting to `False` will reinstall the package into a virtual environment, even if it
1519        is installed outside.
1520
1521    color: bool, default True
1522        If `False`, do not print ANSI colors.
1523
1524    Returns
1525    -------
1526    The specified modules. If they're not available and `install` is `True`, it will first
1527    download them into a virtual environment and return the modules.
1528
1529    Examples
1530    --------
1531    >>> pandas, sqlalchemy = attempt_import('pandas', 'sqlalchemy')
1532    >>> pandas = attempt_import('pandas')
1533
1534    """
1535
1536    import importlib.util
1537
1538    global emitted_auto_install_warning
1539    no_auto_install_env_var = NO_AUTO_INSTALL_ENV_VAR
1540    if not _auto_install_allowed():
1541        install = False
1542
1543    ### Cosmetic packages are never installed on import: without them output is plain text.
1544    if any(name.split('.')[0] in _NEVER_AUTO_INSTALLED for name in names):
1545        install = False
1546        warn = False
1547
1548    ### to prevent recursion, check if parent Meerschaum package is being imported
1549    if names == ('meerschaum',):
1550        return _import_module('meerschaum')
1551
1552    if venv == 'mrsm' and _import_hook_venv is not None:
1553        if debug:
1554            print(f"Import hook for virtual environment '{_import_hook_venv}' is active.")
1555        venv = _import_hook_venv
1556
1557    _warnings = _import_module('meerschaum.utils.warnings')
1558    warn_function = _warnings.warn
1559
1560    def do_import(_name: str, **kw) -> Union['ModuleType', None]:
1561        ### A fully-imported module resolves from `sys.modules` regardless of the active venv,
1562        ### so skip activating the venv (which resolves and creates its directories).
1563        import sys
1564        loaded_module = sys.modules.get(_name, None)
1565        if (
1566            loaded_module is not None
1567            and not getattr(getattr(loaded_module, '__spec__', None), '_initializing', False)
1568        ):
1569            return loaded_module
1570
1571        with Venv(venv=venv, debug=debug):
1572            ### determine the import method (lazy vs normal)
1573            from meerschaum.utils.misc import filter_keywords
1574            import_method = (
1575                _import_module if not lazy
1576                else lazy_import
1577            )
1578            try:
1579                mod = import_method(_name, **(filter_keywords(import_method, **kw)))
1580            except Exception as e:
1581                if warn:
1582                    import traceback
1583                    traceback.print_exception(type(e), e, e.__traceback__)
1584                    warn_function(
1585                        f"Failed to import module '{_name}'.\nException:\n{e}",
1586                        ImportWarning,
1587                        stacklevel = (5 if lazy else 4),
1588                        color = False,
1589                    )
1590                mod = None
1591        return mod
1592
1593    modules = []
1594    for name in names:
1595        ### Check if package is a declared dependency.
1596        root_name = name.split('.')[0] if split else name
1597        install_name = _import_to_install_name(root_name)
1598
1599        if install_name is None:
1600            install_name = root_name
1601            if warn and root_name != 'plugins':
1602                warn_function(
1603                    f"Package '{root_name}' is not declared in meerschaum.utils.packages.",
1604                    ImportWarning,
1605                    stacklevel = 3,
1606                    color = False
1607                )
1608
1609        ### Determine if the package exists.
1610        if precheck is False:
1611            found_module = (
1612                do_import(
1613                    name, debug=debug, warn=False, venv=venv, color=color,
1614                    check_update=False, check_pypi=False, split=split,
1615                ) is not None
1616            )
1617        else:
1618            installed_cache_key = (name, venv, split, allow_outside_venv)
1619            if check_is_installed:
1620                with _locks['_is_installed_first_check']:
1621                    if not _is_installed_first_check.get(installed_cache_key, False):
1622                        package_is_installed = is_installed(
1623                            name,
1624                            venv = venv,
1625                            split = split,
1626                            allow_outside_venv = allow_outside_venv,
1627                            debug = debug,
1628                        )
1629                        if package_is_installed:
1630                            _is_installed_first_check[installed_cache_key] = True
1631                    else:
1632                        package_is_installed = True
1633            else:
1634                package_is_installed = venv_contains_package(
1635                    name,
1636                    venv=venv,
1637                    split=split,
1638                    debug=debug,
1639                )
1640            found_module = package_is_installed
1641
1642        if not found_module:
1643            if install:
1644                with _locks['emitted_auto_install_warning']:
1645                    if warn and not emitted_auto_install_warning:
1646                        emitted_auto_install_warning = True
1647                        warn_function(
1648                            "Meerschaum is installing a missing runtime dependency. "
1649                            + f"Set {no_auto_install_env_var}=1 to disable automatic downloads.",
1650                            ImportWarning,
1651                            stacklevel=3,
1652                            color=False,
1653                        )
1654                if not pip_install(
1655                    install_name,
1656                    venv = venv,
1657                    split = False,
1658                    check_update = check_update,
1659                    color = color,
1660                    debug = debug,
1661                    _implicit = True,
1662                ):
1663                    if warn:
1664                        warn_function(
1665                            f"Failed to install '{install_name}'.",
1666                            ImportWarning,
1667                            stacklevel = 3,
1668                            color = False,
1669                        )
1670                else:
1671                    _record_import_request(install_name, venv)
1672            elif warn:
1673                ### Raise a warning if we can't find the package and install = False.
1674                warn_function(
1675                    (f"\n\nMissing package '{name}' from virtual environment '{venv}'; "
1676                     + "some features will not work correctly."
1677                     + (
1678                         f"\n\nUnset {no_auto_install_env_var} to allow package installation.\n"
1679                         if no_auto_install_env_var in os.environ
1680                         else "\n\nSet install=True when calling attempt_import.\n"
1681                     )),
1682                    ImportWarning,
1683                    stacklevel = 3,
1684                    color = False,
1685                )
1686
1687        elif found_module:
1688            _record_import_request(install_name, venv, only_plugins=True)
1689
1690        ### Do the import. Will be lazy if lazy=True.
1691        m = do_import(
1692            name, debug=debug, warn=warn, venv=venv, color=color,
1693            check_update=check_update, check_pypi=check_pypi, install=install, split=split,
1694        )
1695        modules.append(m)
1696
1697    modules = tuple(modules)
1698    if len(modules) == 1:
1699        return modules[0]
1700    return modules

Raise a warning if packages are not installed; otherwise import and return modules. If lazy is True, return lazy-imported modules.

Returns tuple of modules if multiple names are provided, else returns one module.

Parameters
  • names (List[str]): The packages to be imported.
  • lazy (bool, default True): If True, lazily load packages.
  • warn (bool, default True): If True, raise a warning if a package cannot be imported.
  • install (bool, default True): If True, attempt to install a missing package into the designated virtual environment. If check_update is True, install updates if available.
  • venv (Optional[str], default 'mrsm'): The virtual environment in which to search for packages and to install packages into.
  • precheck (bool, default True): If True, attempt to find module before importing (necessary for checking if modules exist and retaining lazy imports), otherwise assume lazy is False.
  • split (bool, default True): If True, split packages' names on '.'.
  • check_update (bool, default False): If True and install is True, install updates if the required minimum version does not match.
  • check_pypi (bool, default False): If True and check_update is True, check PyPI when determining whether an update is required.
  • check_is_installed (bool, default True): If True, check if the package is contained in the virtual environment.
  • allow_outside_venv (bool, default True): If True, search outside of the specified virtual environment if the package cannot be found. Setting to False will reinstall the package into a virtual environment, even if it is installed outside.
  • color (bool, default True): If False, do not print ANSI colors.
Returns
  • The specified modules. If they're not available and install is True, it will first
  • download them into a virtual environment and return the modules.
Examples
>>> pandas, sqlalchemy = attempt_import('pandas', 'sqlalchemy')
>>> pandas = attempt_import('pandas')
class Connector: 
 22class Connector(metaclass=abc.ABCMeta):
 23    """
 24    The base connector class to hold connection attributes.
 25    """
 26
 27    IS_INSTANCE: bool = False
 28
 29    def __init__(
 30        self,
 31        type: Optional[str] = None,
 32        label: Optional[str] = None,
 33        **kw: Any
 34    ):
 35        """
 36        Set the given keyword arguments as attributes.
 37
 38        Parameters
 39        ----------
 40        type: str
 41            The `type` of the connector (e.g. `sql`, `api`, `plugin`).
 42
 43        label: str
 44            The `label` for the connector.
 45
 46
 47        Examples
 48        --------
 49        Run `mrsm edit config` and to edit connectors in the YAML file:
 50
 51        ```yaml
 52        meerschaum:
 53            connections:
 54                {type}:
 55                    {label}:
 56                        ### attributes go here
 57        ```
 58
 59        """
 60        self._original_dict = copy.deepcopy(self.__dict__)
 61        self._set_attributes(type=type, label=label, **kw)
 62
 63        ### NOTE: Override `REQUIRED_ATTRIBUTES` if `uri` is set.
 64        self.verify_attributes(
 65            ['uri']
 66            if 'uri' in self.__dict__
 67            else getattr(self, 'REQUIRED_ATTRIBUTES', None)
 68        )
 69
 70    def _reset_attributes(self):
 71        self.__dict__ = self._original_dict
 72
 73    def _set_attributes(
 74        self,
 75        *args,
 76        inherit_default: bool = True,
 77        **kw: Any
 78    ):
 79        from meerschaum._internal.static import STATIC_CONFIG
 80        from meerschaum.utils.warnings import error
 81
 82        self._attributes = {}
 83
 84        default_label = STATIC_CONFIG['connectors']['default_label']
 85
 86        ### NOTE: Support the legacy method of explicitly passing the type.
 87        label = kw.get('label', None)
 88        if label is None:
 89            if len(args) == 2:
 90                label = args[1]
 91            elif len(args) == 0:
 92                label = None
 93            else:
 94                label = args[0]
 95
 96        if label == 'default':
 97            error(
 98                f"Label cannot be 'default'. Did you mean '{default_label}'?",
 99                InvalidAttributesError,
100            )
101        self.__dict__['label'] = label
102
103        from meerschaum.config import get_config
104        conn_configs = copy.deepcopy(get_config('meerschaum', 'connectors'))
105        connector_config = copy.deepcopy(get_config('system', 'connectors'))
106
107        ### inherit attributes from 'default' if exists
108        if inherit_default:
109            inherit_from = 'default'
110            if self.type in conn_configs and inherit_from in conn_configs[self.type]:
111                _inherit_dict = copy.deepcopy(conn_configs[self.type][inherit_from])
112                self._attributes.update(_inherit_dict)
113
114        ### load user config into self._attributes
115        if self.type in conn_configs and self.label in conn_configs[self.type]:
116            self._attributes.update(conn_configs[self.type][self.label] or {})
117
118        ### load system config into self._sys_config
119        ### (deep copy so future Connectors don't inherit changes)
120        if self.type in connector_config:
121            self._sys_config = copy.deepcopy(connector_config[self.type])
122
123        ### add additional arguments or override configuration
124        self._attributes.update(kw)
125
126        ### Remember what the caller passed: a connector built with its own attributes
127        ### (`get_connector('sql', 'x', flavor=...)`) overrides the configured `sql:x` in the
128        ### actions this process runs (`meerschaum._internal.entry.adhoc_context`), as in v4.
129        self.__dict__['_explicit_attributes'] = {
130            key: value
131            for key, value in kw.items()
132            if key not in ('type', 'label')
133        }
134
135        ### finally, update __dict__ with _attributes.
136        self.__dict__.update(self._attributes)
137
138    def verify_attributes(
139        self,
140        required_attributes: Optional[List[str]] = None,
141        debug: bool = False,
142    ) -> None:
143        """
144        Ensure that the required attributes have been met.
145        
146        The Connector base class checks the minimum requirements.
147        Child classes may enforce additional requirements.
148
149        Parameters
150        ----------
151        required_attributes: Optional[List[str]], default None
152            Attributes to be verified. If `None`, default to `['label']`.
153
154        debug: bool, default False
155            Verbosity toggle.
156
157        Returns
158        -------
159        Don't return anything.
160
161        Raises
162        ------
163        An error if any of the required attributes are missing.
164        """
165        from meerschaum.utils.warnings import error
166        from meerschaum.utils.misc import items_str
167        if required_attributes is None:
168            required_attributes = ['type', 'label']
169
170        missing_attributes = set()
171        for a in required_attributes:
172            if a not in self.__dict__:
173                missing_attributes.add(a)
174        if len(missing_attributes) > 0:
175            error(
176                (
177                    f"Missing {items_str(list(missing_attributes))} "
178                    + f"for connector '{self.type}:{self.label}'."
179                ),
180                InvalidAttributesError,
181                silent=True,
182                stack=False
183            )
184
185
186    def __str__(self):
187        """
188        When cast to a string, return type:label.
189        """
190        return f"{self.type}:{self.label}"
191
192    def __repr__(self):
193        """
194        Represent the connector as type:label.
195        """
196        return str(self)
197
198    @property
199    def meta(self) -> Dict[str, Any]:
200        """
201        Return the keys needed to reconstruct this Connector.
202        """
203        _meta = {
204            key: value
205            for key, value in self.__dict__.items()
206            if not str(key).startswith('_')
207        }
208        _meta.update({
209            'type': self.type,
210            'label': self.label,
211        })
212        return _meta
213
214
215    @property
216    def protected(self) -> bool:
217        """
218        Whether the connector is protected (`protected: true` in its attributes, or
219        `?protected=true` on its URI): every write through it is refused unless writes to
220        protected connectors are allowed (`--allow-protected`, `MRSM_ALLOW_PROTECTED=1`, or
221        `meerschaum.connectors.allow_protected()`). Reads are never blocked.
222        """
223        from meerschaum.connectors import is_protected_value, uri_marks_protected
224        return (
225            is_protected_value(self.__dict__.get('protected', False))
226            or uri_marks_protected(self.__dict__.get('uri', None))
227        )
228
229    @protected.setter
230    def protected(self, value: Any) -> None:
231        self.__dict__['protected'] = value
232
233    @property
234    def type(self) -> str:
235        """
236        Return the type for this connector.
237        """
238        _type = self.__dict__.get('type', None)
239        if _type is None:
240            import re
241            is_executor = self.__class__.__name__.lower().endswith('executor')
242            suffix_regex = (
243                r'connector$'
244                if not is_executor
245                else r'executor$'
246            )
247            _type = re.sub(suffix_regex, '', self.__class__.__name__.lower())
248            if not _type or _type.lower() == 'instance':
249                raise ValueError("No type could be determined for this connector.")
250            self.__dict__['type'] = _type
251        return _type
252
253
254    @property
255    def label(self) -> str:
256        """
257        Return the label for this connector.
258        """
259        _label = self.__dict__.get('label', None)
260        if _label is None:
261            from meerschaum._internal.static import STATIC_CONFIG
262            _label = STATIC_CONFIG['connectors']['default_label']
263            self.__dict__['label'] = _label
264        return _label

The base connector class to hold connection attributes.

Connector(type: str | None = None, label: str | None = None, **kw: Any) 
29    def __init__(
30        self,
31        type: Optional[str] = None,
32        label: Optional[str] = None,
33        **kw: Any
34    ):
35        """
36        Set the given keyword arguments as attributes.
37
38        Parameters
39        ----------
40        type: str
41            The `type` of the connector (e.g. `sql`, `api`, `plugin`).
42
43        label: str
44            The `label` for the connector.
45
46
47        Examples
48        --------
49        Run `mrsm edit config` and to edit connectors in the YAML file:
50
51        ```yaml
52        meerschaum:
53            connections:
54                {type}:
55                    {label}:
56                        ### attributes go here
57        ```
58
59        """
60        self._original_dict = copy.deepcopy(self.__dict__)
61        self._set_attributes(type=type, label=label, **kw)
62
63        ### NOTE: Override `REQUIRED_ATTRIBUTES` if `uri` is set.
64        self.verify_attributes(
65            ['uri']
66            if 'uri' in self.__dict__
67            else getattr(self, 'REQUIRED_ATTRIBUTES', None)
68        )

Set the given keyword arguments as attributes.

Parameters
  • type (str): The type of the connector (e.g. sql, api, plugin).
  • label (str): The label for the connector.
Examples

Run mrsm edit config and to edit connectors in the YAML file:

meerschaum:
    connections:
        {type}:
            {label}:
                ### attributes go here
IS_INSTANCE: bool = False
def verify_attributes( self, required_attributes: List[str] | None = None, debug: bool = False) -> None: 
138    def verify_attributes(
139        self,
140        required_attributes: Optional[List[str]] = None,
141        debug: bool = False,
142    ) -> None:
143        """
144        Ensure that the required attributes have been met.
145        
146        The Connector base class checks the minimum requirements.
147        Child classes may enforce additional requirements.
148
149        Parameters
150        ----------
151        required_attributes: Optional[List[str]], default None
152            Attributes to be verified. If `None`, default to `['label']`.
153
154        debug: bool, default False
155            Verbosity toggle.
156
157        Returns
158        -------
159        Don't return anything.
160
161        Raises
162        ------
163        An error if any of the required attributes are missing.
164        """
165        from meerschaum.utils.warnings import error
166        from meerschaum.utils.misc import items_str
167        if required_attributes is None:
168            required_attributes = ['type', 'label']
169
170        missing_attributes = set()
171        for a in required_attributes:
172            if a not in self.__dict__:
173                missing_attributes.add(a)
174        if len(missing_attributes) > 0:
175            error(
176                (
177                    f"Missing {items_str(list(missing_attributes))} "
178                    + f"for connector '{self.type}:{self.label}'."
179                ),
180                InvalidAttributesError,
181                silent=True,
182                stack=False
183            )

Ensure that the required attributes have been met.

The Connector base class checks the minimum requirements. Child classes may enforce additional requirements.

Parameters
  • required_attributes (Optional[List[str]], default None): Attributes to be verified. If None, default to ['label'].
  • debug (bool, default False): Verbosity toggle.
Returns
  • Don't return anything.
Raises
  • An error if any of the required attributes are missing.
meta: Dict[str, Any] 
198    @property
199    def meta(self) -> Dict[str, Any]:
200        """
201        Return the keys needed to reconstruct this Connector.
202        """
203        _meta = {
204            key: value
205            for key, value in self.__dict__.items()
206            if not str(key).startswith('_')
207        }
208        _meta.update({
209            'type': self.type,
210            'label': self.label,
211        })
212        return _meta

Return the keys needed to reconstruct this Connector.

protected: bool 
215    @property
216    def protected(self) -> bool:
217        """
218        Whether the connector is protected (`protected: true` in its attributes, or
219        `?protected=true` on its URI): every write through it is refused unless writes to
220        protected connectors are allowed (`--allow-protected`, `MRSM_ALLOW_PROTECTED=1`, or
221        `meerschaum.connectors.allow_protected()`). Reads are never blocked.
222        """
223        from meerschaum.connectors import is_protected_value, uri_marks_protected
224        return (
225            is_protected_value(self.__dict__.get('protected', False))
226            or uri_marks_protected(self.__dict__.get('uri', None))
227        )

Whether the connector is protected (protected: true in its attributes, or ?protected=true on its URI): every write through it is refused unless writes to protected connectors are allowed (--allow-protected, MRSM_ALLOW_PROTECTED=1, or meerschaum.connectors.allow_protected()). Reads are never blocked.

type: str 
233    @property
234    def type(self) -> str:
235        """
236        Return the type for this connector.
237        """
238        _type = self.__dict__.get('type', None)
239        if _type is None:
240            import re
241            is_executor = self.__class__.__name__.lower().endswith('executor')
242            suffix_regex = (
243                r'connector$'
244                if not is_executor
245                else r'executor$'
246            )
247            _type = re.sub(suffix_regex, '', self.__class__.__name__.lower())
248            if not _type or _type.lower() == 'instance':
249                raise ValueError("No type could be determined for this connector.")
250            self.__dict__['type'] = _type
251        return _type

Return the type for this connector.

label: str 
254    @property
255    def label(self) -> str:
256        """
257        Return the label for this connector.
258        """
259        _label = self.__dict__.get('label', None)
260        if _label is None:
261            from meerschaum._internal.static import STATIC_CONFIG
262            _label = STATIC_CONFIG['connectors']['default_label']
263            self.__dict__['label'] = _label
264        return _label

Return the label for this connector.

class InstanceConnector(meerschaum.Connector): 
 18class InstanceConnector(Connector):
 19    """
 20    Instance connectors define the interface for managing pipes and provide methods
 21    for management of users, plugins, tokens, and other metadata built atop pipes.
 22    """
 23
 24    IS_INSTANCE: bool = True
 25    IS_THREAD_SAFE: bool = False
 26
 27    from ._users import (
 28        get_users_pipe,
 29        register_user,
 30        get_user_id,
 31        get_username,
 32        get_users,
 33        edit_user,
 34        delete_user,
 35        get_user_password_hash,
 36        get_user_type,
 37        get_user_attributes,
 38    )
 39
 40    from ._plugins import (
 41        get_plugins_pipe,
 42        register_plugin,
 43        get_plugin_user_id,
 44        delete_plugin,
 45        get_plugin_id,
 46        get_plugin_version,
 47        get_plugins,
 48        get_plugin_user_id,
 49        get_plugin_username,
 50        get_plugin_attributes,
 51    )
 52
 53    from ._tokens import (
 54        get_tokens_pipe,
 55        register_token,
 56        edit_token,
 57        invalidate_token,
 58        delete_token,
 59        get_token,
 60        get_tokens,
 61        get_token_model,
 62        get_token_secret_hash,
 63        token_exists,
 64        get_token_scopes,
 65    )
 66
 67    from ._pipes import (
 68        register_pipe,
 69        get_pipe_attributes,
 70        get_pipe_id,
 71        edit_pipe,
 72        delete_pipe,
 73        fetch_pipes_keys,
 74        pipe_exists,
 75        drop_pipe,
 76        drop_pipe_indices,
 77        sync_pipe,
 78        create_pipe_indices,
 79        clear_pipe,
 80        get_pipe_data,
 81        get_pipe_docs,
 82        get_sync_time,
 83        get_pipe_columns_types,
 84        get_pipe_columns_indices,
 85        get_pipe_size,
 86        compress_pipe,
 87        decompress_pipe,
 88        vacuum_pipe,
 89        analyze_pipe,
 90        partition_pipe,
 91    )
 92
 93    ### Teams, pipe locks and pipe access (the access model, `meerschaum.core.access`).
 94    from ._teams import (
 95        get_teams,
 96        get_team,
 97        register_team,
 98        edit_team,
 99        delete_team,
100        rename_team,
101        get_user_teams,
102        get_team_pipes_count,
103        get_user_permissions,
104        get_user_role,
105        set_user_attributes,
106        lock_pipe,
107        unlock_pipe,
108        set_pipe_access,
109    )

Instance connectors define the interface for managing pipes and provide methods for management of users, plugins, tokens, and other metadata built atop pipes.

IS_INSTANCE: bool = True
IS_THREAD_SAFE: bool = False
def get_users_pipe(self) -> Pipe: 
18def get_users_pipe(self) -> 'mrsm.Pipe':
19    """
20    Return the pipe used for users registration.
21    """
22    if '_users_pipe' in self.__dict__:
23        return self._users_pipe
24
25    cache_connector = self.__dict__.get('_cache_connector', None)
26    self._users_pipe = mrsm.Pipe(
27        'mrsm', 'users',
28        instance=self,
29        target='mrsm_users',
30        temporary=True,
31        cache=True,
32        cache_connector_keys=cache_connector,
33        static=True,
34        null_indices=False,
35        columns={
36            'primary': 'user_id',
37        },
38        dtypes={
39            'user_id': 'uuid',
40            'username': 'string',
41            'password_hash': 'string',
42            'email': 'string',
43            'user_type': 'string',
44            'attributes': 'json',
45        },
46        indices={
47            'unique': 'username',
48        },
49    )
50    return self._users_pipe

Return the pipe used for users registration.

def register_user( self, user: meerschaum.core.User._User.User, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
53def register_user(
54    self,
55    user: User,
56    debug: bool = False,
57    **kwargs: Any
58) -> mrsm.SuccessTuple:
59    """
60    Register a new user to the users pipe.
61    """
62    users_pipe = self.get_users_pipe()
63    user.user_id = uuid.uuid4()
64    sync_success, sync_msg = users_pipe.sync(
65        [{
66            'user_id': user.user_id,
67            'username': user.username,
68            'email': user.email,
69            'password_hash': user.password_hash,
70            'user_type': user.type,
71            'attributes': user.attributes,
72        }],
73        check_existing=False,
74        debug=debug,
75    )
76    if not sync_success:
77        return False, f"Failed to register user '{user.username}':\n{sync_msg}"
78
79    return True, "Success"

Register a new user to the users pipe.

def get_user_id( self, user: meerschaum.core.User._User.User, debug: bool = False) -> uuid.UUID | None: 
142def get_user_id(self, user: User, debug: bool = False) -> Union[uuid.UUID, None]:
143    """
144    Return a user's ID from the username (matched exactly).
145    """
146    users_pipe = self.get_users_pipe()
147    row = _exact_row(users_pipe, 'username', user.username, ['user_id'], debug=debug)
148    return row.get('user_id', None) if row else None

Return a user's ID from the username (matched exactly).

def get_username(self, user_id: Any, debug: bool = False) -> Any: 
151def get_username(self, user_id: Any, debug: bool = False) -> Any:
152    """
153    Return the username from the given ID (matched exactly).
154    """
155    users_pipe = self.get_users_pipe()
156    row = _exact_row(users_pipe, 'user_id', user_id, ['username'], debug=debug)
157    return row.get('username', None) if row else None

Return the username from the given ID (matched exactly).

def get_users(self, debug: bool = False, **kw: Any) -> List[str]: 
172def get_users(
173    self,
174    debug: bool = False,
175    **kw: Any
176) -> List[str]:
177    """
178    Get the registered usernames.
179    """
180    users_pipe = self.get_users_pipe()
181    df = users_pipe.get_data()
182    if df is None:
183        return []
184
185    return list(df['username'])

Get the registered usernames.

def edit_user( self, user: meerschaum.core.User._User.User, debug: bool = False) -> Tuple[bool, str]: 
188def edit_user(self, user: User, debug: bool = False) -> mrsm.SuccessTuple:
189    """
190    Edit the attributes for an existing user.
191    """
192    users_pipe = self.get_users_pipe()
193    user_id = _resolve_user_id(self, user, debug=debug)
194    if user_id is None:
195        return False, (
196            f"User '{user.username}' does not exist. "
197            f"Register user '{user.username}' before editing."
198        )
199
200    doc = {'user_id': user_id}
201    if user.email != '':
202        doc['email'] = user.email
203    if user.password_hash != '':
204        doc['password_hash'] = user.password_hash
205    if user.type != '':
206        doc['user_type'] = user.type
207    if user.attributes:
208        doc['attributes'] = user.attributes
209
210    sync_success, sync_msg = users_pipe.sync([doc], debug=debug)
211    if not sync_success:
212        return False, f"Failed to edit user '{user.username}':\n{sync_msg}"
213
214    return True, "Success"

Edit the attributes for an existing user.

def delete_user( self, user: meerschaum.core.User._User.User, debug: bool = False) -> Tuple[bool, str]: 
217def delete_user(self, user: User, debug: bool = False) -> mrsm.SuccessTuple:
218    """
219    Delete a user from the users table.
220    """
221    user_id = _resolve_user_id(self, user, debug=debug)
222    if user_id is None:
223        return False, f"User '{user.username}' is not registered and cannot be deleted."
224    if not _literal_param_safe(str(user_id)):
225        return False, f"Cannot delete user '{user.username}': its ID cannot be matched exactly."
226    users_pipe = self.get_users_pipe()
227    clear_success, clear_msg = users_pipe.clear(params={'user_id': user_id}, debug=debug)
228    if not clear_success:
229        return False, f"Failed to delete user '{user}':\n{clear_msg}"
230    ### The user leaves every team; their pipes keep their team, without an owner.
231    try:
232        from meerschaum import _core
233        from meerschaum.connectors.instance._teams import _carrier
234        carrier, context = _carrier(self)
235        forgotten, forgotten_msg = _core.forget_user_access(
236            carrier, context, user.username, debug=debug,
237        )
238    except Exception as e:
239        forgotten, forgotten_msg = False, str(e)
240    if not forgotten:
241        from meerschaum.utils.warnings import warn
242        warn(forgotten_msg, stack=False)
243    return True, "Success"

Delete a user from the users table.

def get_user_password_hash( self, user: meerschaum.core.User._User.User, debug: bool = False) -> uuid.UUID | None: 
246def get_user_password_hash(self, user: User, debug: bool = False) -> Union[uuid.UUID, None]:
247    """
248    Get a user's password hash from the users table.
249    """
250    return _user_field(self, user, 'password_hash', debug=debug)

Get a user's password hash from the users table.

def get_user_type( self, user: meerschaum.core.User._User.User, debug: bool = False) -> str | None: 
253def get_user_type(self, user: User, debug: bool = False) -> Union[str, None]:
254    """
255    Get a user's type from the users table.
256    """
257    return _user_field(self, user, 'user_type', debug=debug)

Get a user's type from the users table.

def get_user_attributes( self, user: meerschaum.core.User._User.User, debug: bool = False) -> Dict[str, Any] | None: 
260def get_user_attributes(self, user: User, debug: bool = False) -> Union[Dict[str, Any], None]:
261    """
262    Get a user's attributes from the users table.
263    """
264    return _user_field(self, user, 'attributes', debug=debug)

Get a user's attributes from the users table.

def get_plugins_pipe(self) -> Pipe: 
16def get_plugins_pipe(self) -> 'mrsm.Pipe':
17    """
18    Return the internal pipe for syncing plugins metadata.
19    """
20    if '_plugins_pipe' in self.__dict__:
21        return self._plugins_pipe
22
23    cache_connector = self.__dict__.get('_cache_connector', None)
24    users_pipe = self.get_users_pipe()
25    user_id_dtype = users_pipe.dtypes.get('user_id', 'uuid')
26
27    self._plugins_pipe = mrsm.Pipe(
28        'mrsm', 'plugins',
29        instance=self,
30        target='mrsm_plugins',
31        temporary=True,
32        cache=True,
33        cache_connector_keys=cache_connector,
34        static=True,
35        null_indices=False,
36        columns={
37            'primary': 'plugin_name',
38            'user_id': 'user_id',
39        },
40        dtypes={
41            'plugin_name': 'string',
42            'user_id': user_id_dtype,
43            'attributes': 'json',
44            'version': 'string',
45        },
46    )
47    return self._plugins_pipe

Return the internal pipe for syncing plugins metadata.

def register_plugin( self, plugin: Plugin, debug: bool = False) -> Tuple[bool, str]: 
50def register_plugin(self, plugin: Plugin, debug: bool = False) -> mrsm.SuccessTuple:
51    """
52    Register a new plugin to the plugins table.
53    """
54    plugins_pipe = self.get_plugins_pipe()
55    users_pipe = self.get_users_pipe()
56    user_id = self.get_plugin_user_id(plugin)
57    if user_id is not None:
58        username = self.get_username(user_id, debug=debug)
59        return False, f"{plugin} is already registered to '{username}'."
60
61    doc = {
62        'plugin_name': plugin.name,
63        'version': plugin.version,
64        'attributes': plugin.attributes,
65        'user_id': plugin.user_id,
66    }
67
68    sync_success, sync_msg = plugins_pipe.sync(
69        [doc],
70        check_existing=False,
71        debug=debug,
72    )
73    if not sync_success:
74        return False, f"Failed to register {plugin}:\n{sync_msg}"
75
76    return True, "Success"

Register a new plugin to the plugins table.

def get_plugin_user_id( self, plugin: Plugin, debug: bool = False) -> uuid.UUID | None: 
79def get_plugin_user_id(self, plugin: Plugin, debug: bool = False) -> Union[uuid.UUID, None]:
80    """
81    Return the user ID for plugin's owner.
82    """
83    plugins_pipe = self.get_plugins_pipe() 
84    return plugins_pipe.get_value('user_id', {'plugin_name': plugin.name}, debug=debug)

Return the user ID for plugin's owner.

def delete_plugin( self, plugin: Plugin, debug: bool = False) -> Tuple[bool, str]: 
105def delete_plugin(self, plugin: Plugin, debug: bool = False) -> mrsm.SuccessTuple:
106    """
107    Delete a plugin's registration.
108    """
109    plugin_id = self.get_plugin_id(plugin, debug=debug)
110    if plugin_id is None:
111        return False, f"{plugin} is not registered."
112    
113    plugins_pipe = self.get_plugins_pipe()
114    clear_success, clear_msg = plugins_pipe.clear(params={'plugin_name': plugin.name}, debug=debug)
115    if not clear_success:
116        return False, f"Failed to delete {plugin}:\n{clear_msg}"
117    return True, "Success"

Delete a plugin's registration.

def get_plugin_id( self, plugin: Plugin, debug: bool = False) -> str | None: 
 97def get_plugin_id(self, plugin: Plugin, debug: bool = False) -> Union[str, None]:
 98    """
 99    Return a plugin's ID.
100    """
101    user_id = self.get_plugin_user_id(plugin, debug=debug)
102    return plugin.name if user_id is not None else None

Return a plugin's ID.

def get_plugin_version( self, plugin: Plugin, debug: bool = False) -> str | None: 
120def get_plugin_version(self, plugin: Plugin, debug: bool = False) -> Union[str, None]:
121    """
122    Return the version for a plugin.
123    """
124    plugins_pipe = self.get_plugins_pipe() 
125    return plugins_pipe.get_value('version', {'plugin_name': plugin.name}, debug=debug)

Return the version for a plugin.

def get_plugins( self, user_id: int | None = None, search_term: str | None = None, debug: bool = False, **kw: Any) -> List[str]: 
136def get_plugins(
137    self,
138    user_id: Optional[int] = None,
139    search_term: Optional[str] = None,
140    debug: bool = False,
141    **kw: Any
142) -> List[str]:
143    """
144    Return a list of plugin names.
145    """
146    plugins_pipe = self.get_plugins_pipe()
147    params = {}
148    if user_id:
149        params['user_id'] = user_id
150
151    docs = plugins_pipe.get_docs(select_columns=['plugin_name'], params=params, debug=debug)
152    if not docs:
153        return []
154
155    return [
156        plugin_name
157        for doc in docs
158        if (plugin_name := doc['plugin_name']).startswith(search_term or '')
159    ]

Return a list of plugin names.

def get_plugin_username( self, plugin: Plugin, debug: bool = False) -> uuid.UUID | None: 
87def get_plugin_username(self, plugin: Plugin, debug: bool = False) -> Union[uuid.UUID, None]:
88    """
89    Return the username for plugin's owner.
90    """
91    user_id = self.get_plugin_user_id(plugin, debug=debug)
92    if user_id is None:
93        return None
94    return self.get_username(user_id, debug=debug)

Return the username for plugin's owner.

def get_plugin_attributes( self, plugin: Plugin, debug: bool = False) -> Dict[str, Any]: 
128def get_plugin_attributes(self, plugin: Plugin, debug: bool = False) -> Dict[str, Any]:
129    """
130    Return the attributes for a plugin.
131    """
132    plugins_pipe = self.get_plugins_pipe() 
133    return plugins_pipe.get_value('attributes', {'plugin_name': plugin.name}, debug=debug) or {}

Return the attributes for a plugin.

def get_tokens_pipe(self) -> Pipe: 
52def get_tokens_pipe(self) -> mrsm.Pipe:
53    """
54    Return the internal pipe for tokens management.
55    """
56    if '_tokens_pipe' in self.__dict__:
57        return self._tokens_pipe
58
59    users_pipe = self.get_users_pipe()
60    user_id_dtype = (
61        users_pipe._attributes.get('parameters', {}).get('dtypes', {}).get('user_id', 'uuid')
62    )
63
64    cache_connector = self.__dict__.get('_cache_connector', None)
65
66    self._tokens_pipe = mrsm.Pipe(
67        'mrsm', 'tokens',
68        instance=self,
69        target='mrsm_tokens',
70        temporary=True,
71        cache=True,
72        cache_connector_keys=cache_connector,
73        static=True,
74        autotime=True,
75        null_indices=False,
76        columns={
77            'datetime': 'creation',
78            'primary': 'id',
79        },
80        indices={
81            'unique': 'label',
82            'user_id': 'user_id',
83        },
84        dtypes={
85            'id': 'uuid',
86            'creation': 'datetime',
87            'expiration': 'datetime',
88            'is_valid': 'bool',
89            'label': 'string',
90            'user_id': user_id_dtype,
91            'scopes': 'json',
92            'secret_hash': 'string',
93        },
94    )
95    return self._tokens_pipe

Return the internal pipe for tokens management.

def register_token( self, token: meerschaum.core.Token._Token.Token, debug: bool = False) -> Tuple[bool, str]: 
 98def register_token(
 99    self,
100    token: Token,
101    debug: bool = False,
102) -> mrsm.SuccessTuple:
103    """
104    Register the new token to the tokens table.
105    """
106    token_id, token_secret = token.generate_credentials()
107    tokens_pipe = self.get_tokens_pipe()
108    user_id = self.get_user_id(token.user) if token.user is not None else None
109    if user_id is None:
110        return False, "Cannot register a token without a user."
111
112    doc = {
113        'id': token_id,
114        'user_id': user_id,
115        'creation': datetime.now(timezone.utc),
116        'expiration': token.expiration,
117        'label': token.label,
118        'is_valid': token.is_valid,
119        'scopes': list(token.scopes) if token.scopes else [],
120        'secret_hash': hash_password(
121            str(token_secret),
122            rounds=STATIC_CONFIG['tokens']['hash_rounds']
123        ),
124    }
125    sync_success, sync_msg = tokens_pipe.sync([doc], check_existing=False, debug=debug)
126    if not sync_success:
127        return False, f"Failed to register token:\n{sync_msg}"
128    return True, "Success"

Register the new token to the tokens table.

def edit_token( self, token: meerschaum.core.Token._Token.Token, debug: bool = False) -> Tuple[bool, str]: 
131def edit_token(self, token: Token, debug: bool = False) -> mrsm.SuccessTuple:
132    """
133    Persist the token's in-memory state to the tokens pipe.
134    """
135    if not token.id:
136        return False, "Token ID is not set."
137
138    if not token.exists(debug=debug):
139        return False, f"Token {token.id} does not exist."
140
141    if not token.creation:
142        token_model = self.get_token_model(token.id)
143        token.creation = token_model.creation
144
145    tokens_pipe = self.get_tokens_pipe()
146    doc = {
147        'id': token.id,
148        'creation': token.creation,
149        'expiration': token.expiration,
150        'label': token.label,
151        'is_valid': token.is_valid,
152        'scopes': list(token.scopes) if token.scopes else [],
153    }
154    sync_success, sync_msg = tokens_pipe.sync([doc], debug=debug)
155    if not sync_success:
156        return False, f"Failed to edit token '{token.id}':\n{sync_msg}"
157
158    return True, "Success"

Persist the token's in-memory state to the tokens pipe.

def invalidate_token( self, token: meerschaum.core.Token._Token.Token, debug: bool = False) -> Tuple[bool, str]: 
161def invalidate_token(self, token: Token, debug: bool = False) -> mrsm.SuccessTuple:
162    """
163    Set `is_valid` to `False` for the given token.
164    """
165    if not token.id:
166        return False, "Token ID is not set."
167
168    if not token.exists(debug=debug):
169        return False, f"Token {token.id} does not exist."
170
171    if not token.creation:
172        token_model = self.get_token_model(token.id)
173        token.creation = token_model.creation
174
175    token.is_valid = False
176    tokens_pipe = self.get_tokens_pipe()
177    doc = {
178        'id': token.id,
179        'creation': token.creation,
180        'is_valid': False,
181    }
182    sync_success, sync_msg = tokens_pipe.sync([doc], debug=debug)
183    if not sync_success:
184        return False, f"Failed to invalidate token '{token.id}':\n{sync_msg}"
185
186    return True, "Success"

Set is_valid to False for the given token.

def delete_token( self, token: meerschaum.core.Token._Token.Token, debug: bool = False) -> Tuple[bool, str]: 
189def delete_token(self, token: Token, debug: bool = False) -> mrsm.SuccessTuple:
190    """
191    Delete the given token from the tokens table.
192    """
193    if not token.id:
194        return False, "Token ID is not set."
195
196    if not token.exists(debug=debug):
197        return False, f"Token {token.id} does not exist."
198
199    if not token.creation:
200        token_model = self.get_token_model(token.id)
201        token.creation = token_model.creation
202
203    token.is_valid = False
204    tokens_pipe = self.get_tokens_pipe()
205    clear_success, clear_msg = tokens_pipe.clear(params={'id': token.id}, debug=debug)
206    if not clear_success:
207        return False, f"Failed to delete token '{token.id}':\n{clear_msg}"
208
209    return True, "Success"

Delete the given token from the tokens table.

def get_token( self, token_id: uuid.UUID | str, debug: bool = False) -> meerschaum.core.Token._Token.Token | None: 
264def get_token(self, token_id: Union[uuid.UUID, str], debug: bool = False) -> Union[Token, None]:
265    """
266    Return the `Token` from its ID.
267    """
268    from meerschaum.utils.misc import is_uuid
269    if isinstance(token_id, str):
270        if is_uuid(token_id):
271            token_id = uuid.UUID(token_id)
272        else:
273            raise ValueError("Invalid token ID.")
274    token_model = self.get_token_model(token_id)
275    if token_model is None:
276        return None
277    return Token(instance=self, **dict(token_model))

Return the Token from its ID.

def get_tokens( self, user: meerschaum.core.User._User.User | None = None, labels: List[str] | None = None, ids: List[uuid.UUID] | None = None, debug: bool = False) -> List[meerschaum.core.Token._Token.Token]: 
212def get_tokens(
213    self,
214    user: Optional[User] = None,
215    labels: Optional[List[str]] = None,
216    ids: Optional[List[uuid.UUID]] = None,
217    debug: bool = False,
218) -> List[Token]:
219    """
220    Return a list of `Token` objects.
221    """
222    tokens_pipe = self.get_tokens_pipe()
223    user_id = (
224        self.get_user_id(user, debug=debug)
225        if user is not None
226        else None
227    )
228    user_type = self.get_user_type(user, debug=debug) if user is not None else None
229    params = (
230        {
231            'user_id': (
232                user_id
233                if user_type != 'admin'
234                else [user_id, None]
235            )
236        }
237        if user_id is not None
238        else {}
239    )
240    if labels:
241        params['label'] = labels
242    if ids:
243        params['id'] = ids
244        
245    if debug:
246        dprint(f"Getting tokens with {user_id=}, {params=}")
247
248    tokens_docs = tokens_pipe.get_docs(params=params, debug=debug)
249    if not tokens_docs:
250        return []
251
252    if debug:
253        dprint(f"Retrieved token documents:\n{tokens_docs}")
254
255    return [
256        Token(
257            instance=self,
258            **{**token_doc, 'scopes': _scopes_list(token_doc.get('scopes'))}
259        )
260        for token_doc in reversed(tokens_docs)
261    ]

Return a list of Token objects.

def get_token_model( self, token_id: uuid.UUID | meerschaum.core.Token._Token.Token, debug: bool = False) -> "'Union[TokenModel, None]'": 
280def get_token_model(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> 'Union[TokenModel, None]':
281    """
282    Return a token's model from the instance.
283    """
284    from meerschaum.models import TokenModel
285    if isinstance(token_id, Token):
286        token_id = token_id.id
287    if not token_id:
288        raise ValueError("Invalid token ID.")
289    tokens_pipe = self.get_tokens_pipe()
290    doc = tokens_pipe.get_doc(
291        params={'id': token_id},
292        debug=debug,
293    )
294    if doc is None:
295        return None
296    doc = dict(doc)
297    doc['scopes'] = _scopes_list(doc.get('scopes'))
298    return TokenModel(**doc)

Return a token's model from the instance.

def get_token_secret_hash( self, token_id: uuid.UUID | meerschaum.core.Token._Token.Token, debug: bool = False) -> str | None: 
301def get_token_secret_hash(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> Union[str, None]:
302    """
303    Return the secret hash for a given token.
304    """
305    if isinstance(token_id, Token):
306        token_id = token_id.id
307    if not token_id:
308        raise ValueError("Invalid token ID.")
309    tokens_pipe = self.get_tokens_pipe()
310    return tokens_pipe.get_value('secret_hash', params={'id': token_id}, debug=debug)

Return the secret hash for a given token.

def token_exists( self, token_id: uuid.UUID | meerschaum.core.Token._Token.Token, debug: bool = False) -> bool: 
341def token_exists(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> bool:
342    """
343    Return `True` if a token exists in the tokens pipe.
344    """
345    if isinstance(token_id, Token):
346        token_id = token_id.id
347    if not token_id:
348        raise ValueError("Invalid token ID.")
349
350    tokens_pipe = self.get_tokens_pipe()
351    return tokens_pipe.get_value('creation', params={'id': token_id}, debug=debug) is not None

Return True if a token exists in the tokens pipe.

def get_token_scopes( self, token_id: uuid.UUID | meerschaum.core.Token._Token.Token, debug: bool = False) -> List[str]: 
326def get_token_scopes(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> List[str]:
327    """
328    Return the scopes for a token.
329    """
330    if isinstance(token_id, Token):
331        token_id = token_id.id
332    if not token_id:
333        raise ValueError("Invalid token ID.")
334
335    tokens_pipe = self.get_tokens_pipe()
336    return _scopes_list(
337        tokens_pipe.get_value('scopes', params={'id': token_id}, debug=debug)
338    )

Return the scopes for a token.

@abc.abstractmethod
def register_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
17@abc.abstractmethod
18def register_pipe(
19    self,
20    pipe: mrsm.Pipe,
21    debug: bool = False,
22    **kwargs: Any
23) -> mrsm.SuccessTuple:
24    """
25    Insert the pipe's attributes into the internal `pipes` table.
26
27    Parameters
28    ----------
29    pipe: mrsm.Pipe
30        The pipe to be registered.
31
32    Returns
33    -------
34    A `SuccessTuple` of the result.
35    """

Insert the pipe's attributes into the internal pipes table.

Parameters
  • pipe (mrsm.Pipe): The pipe to be registered.
Returns
@abc.abstractmethod
def get_pipe_attributes( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Dict[str, Any]: 
37@abc.abstractmethod
38def get_pipe_attributes(
39    self,
40    pipe: mrsm.Pipe,
41    debug: bool = False,
42    **kwargs: Any
43) -> Dict[str, Any]:
44    """
45    Return the pipe's document from the internal `pipes` table.
46
47    Parameters
48    ----------
49    pipe: mrsm.Pipe
50        The pipe whose attributes should be retrieved.
51
52    Returns
53    -------
54    The document that matches the keys of the pipe.
55    """

Return the pipe's document from the internal pipes table.

Parameters
  • pipe (mrsm.Pipe): The pipe whose attributes should be retrieved.
Returns
  • The document that matches the keys of the pipe.
@abc.abstractmethod
def get_pipe_id( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> str | int | None: 
57@abc.abstractmethod
58def get_pipe_id(
59    self,
60    pipe: mrsm.Pipe,
61    debug: bool = False,
62    **kwargs: Any
63) -> Union[str, int, None]:
64    """
65    Return the `id` for the pipe if it exists.
66
67    Parameters
68    ----------
69    pipe: mrsm.Pipe
70        The pipe whose `id` to fetch.
71
72    Returns
73    -------
74    The `id` for the pipe's document or `None`.
75    """

Return the id for the pipe if it exists.

Parameters
  • pipe (mrsm.Pipe): The pipe whose id to fetch.
Returns
  • The id for the pipe's document or None.
def edit_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
77def edit_pipe(
78    self,
79    pipe: mrsm.Pipe,
80    debug: bool = False,
81    **kwargs: Any
82) -> mrsm.SuccessTuple:
83    """
84    Edit the attributes of the pipe.
85
86    Parameters
87    ----------
88    pipe: mrsm.Pipe
89        The pipe whose in-memory parameters must be persisted.
90
91    Returns
92    -------
93    A `SuccessTuple` indicating success.
94    """
95    raise NotImplementedError

Edit the attributes of the pipe.

Parameters
  • pipe (mrsm.Pipe): The pipe whose in-memory parameters must be persisted.
Returns
def delete_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
 97def delete_pipe(
 98    self,
 99    pipe: mrsm.Pipe,
100    debug: bool = False,
101    **kwargs: Any
102) -> mrsm.SuccessTuple:
103    """
104    Delete a pipe's registration from the `pipes` collection.
105
106    Parameters
107    ----------
108    pipe: mrsm.Pipe
109        The pipe to be deleted.
110
111    Returns
112    -------
113    A `SuccessTuple` indicating success.
114    """
115    raise NotImplementedError

Delete a pipe's registration from the pipes collection.

Parameters
  • pipe (mrsm.Pipe): The pipe to be deleted.
Returns
@abc.abstractmethod
def fetch_pipes_keys( self, connector_keys: List[str] | None = None, metric_keys: List[str] | None = None, location_keys: List[str] | None = None, tags: List[str] | None = None, debug: bool = False, **kwargs: Any) -> List[Tuple[str, str, str]] | List[Tuple[str, str, str, Dict[str, Any] | List[str]]] | Dict[int | str, Tuple[str, str, str]] | Dict[int | str, Tuple[str, str, str, Dict[str, Any] | List[str]]]: 
117@abc.abstractmethod
118def fetch_pipes_keys(
119    self,
120    connector_keys: Optional[List[str]] = None,
121    metric_keys: Optional[List[str]] = None,
122    location_keys: Optional[List[str]] = None,
123    tags: Optional[List[str]] = None,
124    debug: bool = False,
125    **kwargs: Any
126) -> Union[
127    List[Tuple[str, str, str]],
128    List[Tuple[str, str, str, Union[Dict[str, Any], List[str]]]],
129    Dict[Union[int, str], Tuple[str, str, str]],
130    Dict[Union[int, str], Tuple[str, str, str, Union[Dict[str, Any], List[str]]]],
131]:
132    """
133    Return registered pipes' keys according to the provided filters.
134
135    May return either a list of key tuples or a dictionary mapping pipe IDs to key tuples.
136    When returning a dictionary, the key is the pipe's unique ID (int or str).
137    Tuples may be length 3 `(connector_keys, metric_key, location_key)` or length 4
138    with parameters or tags appended as the fourth element.
139
140    Parameters
141    ----------
142    connector_keys: list[str] | None, default None
143        The keys passed via `-c`.
144
145    metric_keys: list[str] | None, default None
146        The keys passed via `-m`.
147
148    location_keys: list[str] | None, default None
149        The keys passed via `-l`.
150
151    tags: List[str] | None, default None
152        Tags passed via `--tags` which are stored under `parameters:tags`.
153
154    Returns
155    -------
156    A list of tuples or a dictionary mapping pipe IDs to tuples.
157    You may return the string `"None"` for location keys in place of nulls.
158
159    Examples
160    --------
161    >>> import meerschaum as mrsm
162    >>> conn = mrsm.get_connector('example:demo')
163    >>>
164    >>> pipe_a = mrsm.Pipe('a', 'demo', tags=['foo'], instance=conn)
165    >>> pipe_b = mrsm.Pipe('b', 'demo', tags=['bar'], instance=conn)
166    >>> pipe_a.register()
167    >>> pipe_b.register()
168    >>>
169    >>> conn.fetch_pipes_keys(['a', 'b'])
170    [('a', 'demo', 'None'), ('b', 'demo', 'None')]
171    >>> conn.fetch_pipes_keys(metric_keys=['demo'])
172    [('a', 'demo', 'None'), ('b', 'demo', 'None')]
173    >>> conn.fetch_pipes_keys(tags=['foo'])
174    [('a', 'demo', 'None')]
175    >>> conn.fetch_pipes_keys(location_keys=[None])
176    [('a', 'demo', 'None'), ('b', 'demo', 'None')]
177    """

Return registered pipes' keys according to the provided filters.

May return either a list of key tuples or a dictionary mapping pipe IDs to key tuples. When returning a dictionary, the key is the pipe's unique ID (int or str). Tuples may be length 3 (connector_keys, metric_key, location_key) or length 4 with parameters or tags appended as the fourth element.

Parameters
  • connector_keys (list[str] | None, default None): The keys passed via -c.
  • metric_keys (list[str] | None, default None): The keys passed via -m.
  • location_keys (list[str] | None, default None): The keys passed via -l.
  • tags (List[str] | None, default None): Tags passed via --tags which are stored under parameters:tags.
Returns
  • A list of tuples or a dictionary mapping pipe IDs to tuples.
  • You may return the string "None" for location keys in place of nulls.
Examples
>>> import meerschaum as mrsm
>>> conn = mrsm.get_connector('example:demo')
>>>
>>> pipe_a = mrsm.Pipe('a', 'demo', tags=['foo'], instance=conn)
>>> pipe_b = mrsm.Pipe('b', 'demo', tags=['bar'], instance=conn)
>>> pipe_a.register()
>>> pipe_b.register()
>>>
>>> conn.fetch_pipes_keys(['a', 'b'])
[('a', 'demo', 'None'), ('b', 'demo', 'None')]
>>> conn.fetch_pipes_keys(metric_keys=['demo'])
[('a', 'demo', 'None'), ('b', 'demo', 'None')]
>>> conn.fetch_pipes_keys(tags=['foo'])
[('a', 'demo', 'None')]
>>> conn.fetch_pipes_keys(location_keys=[None])
[('a', 'demo', 'None'), ('b', 'demo', 'None')]
@abc.abstractmethod
def pipe_exists( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> bool: 
179@abc.abstractmethod
180def pipe_exists(
181    self,
182    pipe: mrsm.Pipe,
183    debug: bool = False,
184    **kwargs: Any
185) -> bool:
186    """
187    Check whether a pipe's target table exists.
188
189    Parameters
190    ----------
191    pipe: mrsm.Pipe
192        The pipe to check whether its table exists.
193
194    Returns
195    -------
196    A `bool` indicating the table exists.
197    """

Check whether a pipe's target table exists.

Parameters
  • pipe (mrsm.Pipe): The pipe to check whether its table exists.
Returns
  • A bool indicating the table exists.
@abc.abstractmethod
def drop_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
199@abc.abstractmethod
200def drop_pipe(
201    self,
202    pipe: mrsm.Pipe,
203    debug: bool = False,
204    **kwargs: Any
205) -> mrsm.SuccessTuple:
206    """
207    Drop a pipe's collection if it exists.
208
209    Parameters
210    ----------
211    pipe: mrsm.Pipe
212        The pipe to be dropped.
213
214    Returns
215    -------
216    A `SuccessTuple` indicating success.
217    """
218    raise NotImplementedError

Drop a pipe's collection if it exists.

Parameters
  • pipe (mrsm.Pipe): The pipe to be dropped.
Returns
def drop_pipe_indices( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
220def drop_pipe_indices(
221    self,
222    pipe: mrsm.Pipe,
223    debug: bool = False,
224    **kwargs: Any
225) -> mrsm.SuccessTuple:
226    """
227    Drop a pipe's indices.
228
229    Parameters
230    ----------
231    pipe: mrsm.Pipe
232        The pipe whose indices need to be dropped.
233
234    Returns
235    -------
236    A `SuccessTuple` indicating success.
237    """
238    return False, f"Cannot drop indices for instance connectors of type '{self.type}'."

Drop a pipe's indices.

Parameters
  • pipe (mrsm.Pipe): The pipe whose indices need to be dropped.
Returns
@abc.abstractmethod
def sync_pipe( self, pipe: Pipe, df: "'pd.DataFrame'" = None, begin: datetime.datetime | int | None = None, end: datetime.datetime | int | None = None, chunksize: int | None = -1, check_existing: bool = True, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
240@abc.abstractmethod
241def sync_pipe(
242    self,
243    pipe: mrsm.Pipe,
244    df: 'pd.DataFrame' = None,
245    begin: Union[datetime, int, None] = None,
246    end: Union[datetime, int, None] = None,
247    chunksize: Optional[int] = -1,
248    check_existing: bool = True,
249    debug: bool = False,
250    **kwargs: Any
251) -> mrsm.SuccessTuple:
252    """
253    Sync a pipe using a database connection.
254
255    Parameters
256    ----------
257    pipe: mrsm.Pipe
258        The Meerschaum Pipe instance into which to sync the data.
259
260    df: Optional[pd.DataFrame]
261        An optional DataFrame or equivalent to sync into the pipe.
262        Defaults to `None`.
263
264    begin: Union[datetime, int, None], default None
265        Optionally specify the earliest datetime to search for data.
266        Defaults to `None`.
267
268    end: Union[datetime, int, None], default None
269        Optionally specify the latest datetime to search for data.
270        Defaults to `None`.
271
272    chunksize: Optional[int], default -1
273        Specify the number of rows to sync per chunk.
274        If `-1`, resort to system configuration (default is `900`).
275        A `chunksize` of `None` will sync all rows in one transaction.
276        Defaults to `-1`.
277
278    check_existing: bool, default True
279        If `True`, pull and diff with existing data from the pipe. Defaults to `True`.
280
281    debug: bool, default False
282        Verbosity toggle. Defaults to False.
283
284    Returns
285    -------
286    A `SuccessTuple` of success (`bool`) and message (`str`).
287    """

Sync a pipe using a database connection.

Parameters
  • pipe (mrsm.Pipe): The Meerschaum Pipe instance into which to sync the data.
  • df (Optional[pd.DataFrame]): An optional DataFrame or equivalent to sync into the pipe. Defaults to None.
  • begin (Union[datetime, int, None], default None): Optionally specify the earliest datetime to search for data. Defaults to None.
  • end (Union[datetime, int, None], default None): Optionally specify the latest datetime to search for data. Defaults to None.
  • chunksize (Optional[int], default -1): Specify the number of rows to sync per chunk. If -1, resort to system configuration (default is 900). A chunksize of None will sync all rows in one transaction. Defaults to -1.
  • check_existing (bool, default True): If True, pull and diff with existing data from the pipe. Defaults to True.
  • debug (bool, default False): Verbosity toggle. Defaults to False.
Returns
def create_pipe_indices( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
289def create_pipe_indices(
290    self,
291    pipe: mrsm.Pipe,
292    debug: bool = False,
293    **kwargs: Any
294) -> mrsm.SuccessTuple:
295    """
296    Create a pipe's indices.
297
298    Parameters
299    ----------
300    pipe: mrsm.Pipe
301        The pipe whose indices need to be created.
302
303    Returns
304    -------
305    A `SuccessTuple` indicating success.
306    """
307    return False, f"Cannot create indices for instance connectors of type '{self.type}'."

Create a pipe's indices.

Parameters
  • pipe (mrsm.Pipe): The pipe whose indices need to be created.
Returns
def clear_pipe( self, pipe: Pipe, begin: datetime.datetime | int | None = None, end: datetime.datetime | int | None = None, params: Dict[str, Any] | None = None, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
309def clear_pipe(
310    self,
311    pipe: mrsm.Pipe,
312    begin: Union[datetime, int, None] = None,
313    end: Union[datetime, int, None] = None,
314    params: Optional[Dict[str, Any]] = None,
315    debug: bool = False,
316    **kwargs: Any
317) -> mrsm.SuccessTuple:
318    """
319    Delete rows within `begin`, `end`, and `params`.
320
321    Parameters
322    ----------
323    pipe: mrsm.Pipe
324        The pipe whose rows to clear.
325
326    begin: datetime | int | None, default None
327        If provided, remove rows >= `begin`.
328
329    end: datetime | int | None, default None
330        If provided, remove rows < `end`.
331
332    params: dict[str, Any] | None, default None
333        If provided, only remove rows which match the `params` filter.
334
335    Returns
336    -------
337    A `SuccessTuple` indicating success.
338    """
339    raise NotImplementedError

Delete rows within begin, end, and params.

Parameters
  • pipe (mrsm.Pipe): The pipe whose rows to clear.
  • begin (datetime | int | None, default None): If provided, remove rows >= begin.
  • end (datetime | int | None, default None): If provided, remove rows < end.
  • params (dict[str, Any] | None, default None): If provided, only remove rows which match the params filter.
Returns
def get_pipe_data( self, pipe: Pipe, select_columns: List[str] | None = None, omit_columns: List[str] | None = None, begin: datetime.datetime | int | None = None, end: datetime.datetime | int | None = None, params: Dict[str, Any] | None = None, debug: bool = False, **kwargs: Any) -> "Union['pd.DataFrame', None]": 
341def get_pipe_data(
342    self,
343    pipe: mrsm.Pipe,
344    select_columns: Optional[List[str]] = None,
345    omit_columns: Optional[List[str]] = None,
346    begin: Union[datetime, int, None] = None,
347    end: Union[datetime, int, None] = None,
348    params: Optional[Dict[str, Any]] = None,
349    debug: bool = False,
350    **kwargs: Any
351) -> Union['pd.DataFrame', None]:
352    """
353    Query a pipe's target table and return the DataFrame.
354
355    Parameters
356    ----------
357    pipe: mrsm.Pipe
358        The pipe with the target table from which to read.
359
360    select_columns: list[str] | None, default None
361        If provided, only select these given columns.
362        Otherwise select all available columns (i.e. `SELECT *`).
363
364    omit_columns: list[str] | None, default None
365        If provided, remove these columns from the selection.
366
367    begin: datetime | int | None, default None
368        The earliest `datetime` value to search from (inclusive).
369
370    end: datetime | int | None, default None
371        The lastest `datetime` value to search from (exclusive).
372
373    params: dict[str | str] | None, default None
374        Additional filters to apply to the query.
375
376    Returns
377    -------
378    The target table's data as a DataFrame.
379    """
380    if type(self).get_pipe_docs is get_pipe_docs:
381        raise NotImplementedError(
382            f"Missing `get_pipe_data()` or `get_pipe_docs()` for {type(self)}."
383        )
384
385    docs = self.get_pipe_docs(
386        pipe=pipe,
387        select_columns=select_columns,
388        omit_columns=omit_columns,
389        begin=begin,
390        end=end,
391        params=params,
392        debug=debug,
393        **kwargs
394    )
395    if not docs:
396        return None
397
398    pd = mrsm.attempt_import('pandas')
399    try:
400        return pd.DataFrame(docs)
401    except Exception as e:
402        from meerschaum.utils.warnings import warn
403        warn(f"Cannot build DataFrame from pipe docs:\n{e}")
404    
405    return None

Query a pipe's target table and return the DataFrame.

Parameters
  • pipe (mrsm.Pipe): The pipe with the target table from which to read.
  • select_columns (list[str] | None, default None): If provided, only select these given columns. Otherwise select all available columns (i.e. SELECT *).
  • omit_columns (list[str] | None, default None): If provided, remove these columns from the selection.
  • begin (datetime | int | None, default None): The earliest datetime value to search from (inclusive).
  • end (datetime | int | None, default None): The lastest datetime value to search from (exclusive).
  • params (dict[str | str] | None, default None): Additional filters to apply to the query.
Returns
  • The target table's data as a DataFrame.
def get_pipe_docs( self, pipe: Pipe, select_columns: List[str] | None = None, omit_columns: List[str] | None = None, begin: datetime.datetime | int | None = None, end: datetime.datetime | int | None = None, params: Dict[str, Any] | None = None, debug: bool = False, **kwargs: Any) -> list[dict[str, typing.Any]]: 
407def get_pipe_docs(
408    self,
409    pipe: mrsm.Pipe,
410    select_columns: Optional[List[str]] = None,
411    omit_columns: Optional[List[str]] = None,
412    begin: Union[datetime, int, None] = None,
413    end: Union[datetime, int, None] = None,
414    params: Optional[Dict[str, Any]] = None,
415    debug: bool = False,
416    **kwargs: Any
417) -> list[dict[str, Any]]:
418    """
419    Return a pipe's data as a list of documents.
420    Converts either a Polars or pandas frame returned by `get_pipe_data()`.
421
422    Parameters
423    ----------
424    pipe: mrsm.Pipe
425        The pipe with the target table from which to read.
426
427    select_columns: list[str] | None, default None
428        If provided, only select these given columns.
429        Otherwise select all available columns (i.e. `SELECT *`).
430
431    omit_columns: list[str] | None, default None
432        If provided, remove these columns from the selection.
433
434    begin: datetime | int | None, default None
435        The earliest `datetime` value to search from (inclusive).
436
437    end: datetime | int | None, default None
438        The lastest `datetime` value to search from (exclusive).
439
440    params: dict[str | str] | None, default None
441        Additional filters to apply to the query.
442
443    Returns
444    -------
445    The target table's data as a list of dictionaries.
446    """
447    df = self.get_pipe_data(
448        pipe=pipe,
449        select_columns=select_columns,
450        omit_columns=omit_columns,
451        begin=begin,
452        end=end,
453        params=params,
454        debug=debug,
455        **kwargs
456    )
457    if df is None:
458        return []
459    to_dicts = getattr(df, 'to_dicts', None)
460    if callable(to_dicts):
461        return to_dicts()
462    return df.to_dict(orient='records')

Return a pipe's data as a list of documents. Converts either a Polars or pandas frame returned by get_pipe_data().

Parameters
  • pipe (mrsm.Pipe): The pipe with the target table from which to read.
  • select_columns (list[str] | None, default None): If provided, only select these given columns. Otherwise select all available columns (i.e. SELECT *).
  • omit_columns (list[str] | None, default None): If provided, remove these columns from the selection.
  • begin (datetime | int | None, default None): The earliest datetime value to search from (inclusive).
  • end (datetime | int | None, default None): The lastest datetime value to search from (exclusive).
  • params (dict[str | str] | None, default None): Additional filters to apply to the query.
Returns
  • The target table's data as a list of dictionaries.
@abc.abstractmethod
def get_sync_time( self, pipe: Pipe, params: Dict[str, Any] | None = None, newest: bool = True, debug: bool = False, **kwargs: Any) -> datetime.datetime | int | None: 
464@abc.abstractmethod
465def get_sync_time(
466    self,
467    pipe: mrsm.Pipe,
468    params: Optional[Dict[str, Any]] = None,
469    newest: bool = True,
470    debug: bool = False,
471    **kwargs: Any
472) -> datetime | int | None:
473    """
474    Return the most recent value for the `datetime` axis.
475
476    Parameters
477    ----------
478    pipe: mrsm.Pipe
479        The pipe whose collection contains documents.
480
481    params: dict[str, Any] | None, default None
482        Filter certain parameters when determining the sync time.
483
484    newest: bool, default True
485        If `True`, return the maximum value for the column.
486
487    Returns
488    -------
489    The largest `datetime` or `int` value of the `datetime` axis. 
490    """

Return the most recent value for the datetime axis.

Parameters
  • pipe (mrsm.Pipe): The pipe whose collection contains documents.
  • params (dict[str, Any] | None, default None): Filter certain parameters when determining the sync time.
  • newest (bool, default True): If True, return the maximum value for the column.
Returns
  • The largest datetime or int value of the datetime axis.
@abc.abstractmethod
def get_pipe_columns_types( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Dict[str, str]: 
492@abc.abstractmethod
493def get_pipe_columns_types(
494    self,
495    pipe: mrsm.Pipe,
496    debug: bool = False,
497    **kwargs: Any
498) -> Dict[str, str]:
499    """
500    Return the data types for the columns in the target table for data type enforcement.
501
502    Parameters
503    ----------
504    pipe: mrsm.Pipe
505        The pipe whose target table contains columns and data types.
506
507    Returns
508    -------
509    A dictionary mapping columns to data types.
510    """

Return the data types for the columns in the target table for data type enforcement.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table contains columns and data types.
Returns
  • A dictionary mapping columns to data types.
def get_pipe_columns_indices(self, debug: bool = False) -> Dict[str, List[Dict[str, str]]]: 
512def get_pipe_columns_indices(
513    self,
514    debug: bool = False,
515) -> Dict[str, List[Dict[str, str]]]:
516    """
517    Return a dictionary mapping columns to metadata about related indices.
518
519    Parameters
520    ----------
521    pipe: mrsm.Pipe
522        The pipe whose target table has related indices.
523
524    Returns
525    -------
526    A list of dictionaries with the keys "type" and "name".
527
528    Examples
529    --------
530    >>> pipe = mrsm.Pipe('demo', 'shirts', columns={'primary': 'id'}, indices={'size_color': ['color', 'size']})
531    >>> pipe.sync([{'color': 'red', 'size': 'M'}])
532    >>> pipe.get_columns_indices()
533    {'id': [{'name': 'demo_shirts_pkey', 'type': 'PRIMARY KEY'}], 'color': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}], 'size': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}]}
534    """
535    return {}

Return a dictionary mapping columns to metadata about related indices.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table has related indices.
Returns
  • A list of dictionaries with the keys "type" and "name".
Examples
>>> pipe = mrsm.Pipe('demo', 'shirts', columns={'primary': 'id'}, indices={'size_color': ['color', 'size']})
>>> pipe.sync([{'color': 'red', 'size': 'M'}])
>>> pipe.get_columns_indices()
{'id': [{'name': 'demo_shirts_pkey', 'type': 'PRIMARY KEY'}], 'color': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}], 'size': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}]}
def get_pipe_size( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> int | None: 
537def get_pipe_size(
538    self,
539    pipe: mrsm.Pipe,
540    debug: bool = False,
541    **kwargs: Any
542) -> Union[int, None]:
543    """
544    Return the on-disk size of a pipe's target table in bytes.
545
546    Parameters
547    ----------
548    pipe: mrsm.Pipe
549        The pipe whose target table size to measure.
550
551    Returns
552    -------
553    An `int` of the number of bytes occupied by the target table,
554    or `None` if the size cannot be determined.
555    """
556    raise NotImplementedError(
557        f"`get_pipe_size()` is not implemented for instance connectors of type '{self.type}'."
558    )

Return the on-disk size of a pipe's target table in bytes.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table size to measure.
Returns
  • An int of the number of bytes occupied by the target table,
  • or None if the size cannot be determined.
def compress_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
560def compress_pipe(
561    self,
562    pipe: mrsm.Pipe,
563    debug: bool = False,
564    **kwargs: Any
565) -> mrsm.SuccessTuple:
566    """
567    Compress a pipe's target table to reduce disk usage.
568
569    Parameters
570    ----------
571    pipe: mrsm.Pipe
572        The pipe whose target table to compress.
573
574    Returns
575    -------
576    A `SuccessTuple` indicating success.
577    """
578    return False, (
579        f"Compression is not supported for instance connectors of type '{self.type}'."
580    )

Compress a pipe's target table to reduce disk usage.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table to compress.
Returns
def decompress_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
582def decompress_pipe(
583    self,
584    pipe: mrsm.Pipe,
585    debug: bool = False,
586    **kwargs: Any
587) -> mrsm.SuccessTuple:
588    """
589    Decompress a pipe's target table, the inverse of `compress_pipe()`.
590
591    Parameters
592    ----------
593    pipe: mrsm.Pipe
594        The pipe whose target table to decompress.
595
596    Returns
597    -------
598    A `SuccessTuple` indicating success.
599    """
600    return False, (
601        f"Decompression is not supported for instance connectors of type '{self.type}'."
602    )

Decompress a pipe's target table, the inverse of compress_pipe().

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table to decompress.
Returns
def vacuum_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
604def vacuum_pipe(
605    self,
606    pipe: mrsm.Pipe,
607    debug: bool = False,
608    **kwargs: Any
609) -> mrsm.SuccessTuple:
610    """
611    Reclaim disk space from a pipe's target table.
612
613    Parameters
614    ----------
615    pipe: mrsm.Pipe
616        The pipe whose target table to vacuum.
617
618    Returns
619    -------
620    A `SuccessTuple` indicating success.
621    """
622    return False, (
623        f"Vacuuming is not supported for instance connectors of type '{self.type}'."
624    )

Reclaim disk space from a pipe's target table.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table to vacuum.
Returns
def analyze_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
626def analyze_pipe(
627    self,
628    pipe: mrsm.Pipe,
629    debug: bool = False,
630    **kwargs: Any
631) -> mrsm.SuccessTuple:
632    """
633    Refresh the planner statistics for a pipe's target table.
634
635    Parameters
636    ----------
637    pipe: mrsm.Pipe
638        The pipe whose target table to analyze.
639
640    Returns
641    -------
642    A `SuccessTuple` indicating success.
643    """
644    return False, (
645        f"Analyzing is not supported for instance connectors of type '{self.type}'."
646    )

Refresh the planner statistics for a pipe's target table.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table to analyze.
Returns
def partition_pipe( self, pipe: Pipe, chunk_minutes: int | None = None, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
648def partition_pipe(
649    self,
650    pipe: mrsm.Pipe,
651    chunk_minutes: Optional[int] = None,
652    debug: bool = False,
653    **kwargs: Any
654) -> mrsm.SuccessTuple:
655    """
656    Rebuild a pipe's target table to a new partition (chunk) width.
657
658    Parameters
659    ----------
660    pipe: mrsm.Pipe
661        The partitioned pipe whose target table to repartition.
662
663    chunk_minutes: Optional[int], default None
664        The new partition width in minutes. Defaults to the pipe's `verify.chunk_minutes`.
665
666    Returns
667    -------
668    A `SuccessTuple` indicating success.
669    """
670    return False, (
671        f"Repartitioning is not supported for instance connectors of type '{self.type}'."
672    )

Rebuild a pipe's target table to a new partition (chunk) width.

Parameters
  • pipe (mrsm.Pipe): The partitioned pipe whose target table to repartition.
  • chunk_minutes (Optional[int], default None): The new partition width in minutes. Defaults to the pipe's verify.chunk_minutes.
Returns
def get_teams(self, debug: bool = False, **kwargs: Any) -> List[Dict[str, Any]]: 
41def get_teams(self, debug: bool = False, **kwargs: Any) -> List[Dict[str, Any]]:
42    """
43    Return every team on this instance, `Everyone` first.
44    """
45    from meerschaum import _core
46    carrier, context = _carrier(self)
47    return list(_core.get_teams(carrier, context, debug=debug))

Return every team on this instance, Everyone first.

def get_team( self, name: str, debug: bool = False, **kwargs: Any) -> Dict[str, Any] | None: 
50def get_team(self, name: str, debug: bool = False, **kwargs: Any) -> Optional[Dict[str, Any]]:
51    """
52    Return the team named `name` (any case), or `None`.
53    """
54    from meerschaum import _core
55    carrier, context = _carrier(self)
56    return _core.get_team(carrier, context, str(name), debug=debug)

Return the team named name (any case), or None.

def register_team(self, team: Any, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
59def register_team(self, team: Any, debug: bool = False, **kwargs: Any) -> mrsm.SuccessTuple:
60    """
61    Register a new team (a `Team` or a dictionary; its new id is written back to it).
62    """
63    from meerschaum import _core
64    carrier, context = _carrier(self)
65    record = _team_record(team)
66    success, message, registered = _core.register_team(carrier, context, record, debug=debug)
67    if success:
68        new_id = (registered or {}).get('id', None)
69        if isinstance(team, dict):
70            team['id'] = new_id
71        elif hasattr(team, '_id'):
72            team._id = new_id
73    return success, message

Register a new team (a Team or a dictionary; its new id is written back to it).

def edit_team(self, team: Any, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
76def edit_team(self, team: Any, debug: bool = False, **kwargs: Any) -> mrsm.SuccessTuple:
77    """
78    Save a team's members and attributes.
79    """
80    from meerschaum import _core
81    carrier, context = _carrier(self)
82    return tuple(_core.edit_team(carrier, context, _team_record(team), debug=debug))

Save a team's members and attributes.

def delete_team(self, name: Any, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
85def delete_team(self, name: Any, debug: bool = False, **kwargs: Any) -> mrsm.SuccessTuple:
86    """
87    Delete a team: its pipes and saved Pipelines move to `Everyone`.
88    """
89    from meerschaum import _core
90    carrier, context = _carrier(self)
91    return tuple(_core.delete_team(carrier, context, str(getattr(name, 'name', name)), debug=debug))

Delete a team: its pipes and saved Pipelines move to Everyone.

def rename_team( self, old_name: Any, new_name: str, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
 94def rename_team(
 95    self,
 96    old_name: Any,
 97    new_name: str,
 98    debug: bool = False,
 99    **kwargs: Any
100) -> mrsm.SuccessTuple:
101    """
102    Rename a team, in the pipes, saved Pipelines and primary teams that name it too.
103    """
104    from meerschaum import _core
105    carrier, context = _carrier(self)
106    return tuple(_core.rename_team(
107        carrier, context, str(getattr(old_name, 'name', old_name)), str(new_name), debug=debug,
108    ))

Rename a team, in the pipes, saved Pipelines and primary teams that name it too.

def get_user_teams( self, user: Any, debug: bool = False, **kwargs: Any) -> List[Tuple[str, str]]: 
111def get_user_teams(self, user: Any, debug: bool = False, **kwargs: Any) -> List[Tuple[str, str]]:
112    """
113    Return a user's teams as `(team, role)` pairs (`Everyone` first).
114    """
115    from meerschaum import _core
116    carrier, context = _carrier(self)
117    return [tuple(pair) for pair in _core.get_user_teams(carrier, context, _username(user), debug=debug)]

Return a user's teams as (team, role) pairs (Everyone first).

def get_team_pipes_count(self, name: str, debug: bool = False, **kwargs: Any) -> int: 
120def get_team_pipes_count(self, name: str, debug: bool = False, **kwargs: Any) -> int:
121    """
122    Return how many pipes (and saved Pipelines) name a team (what renaming or deleting it
123    changes).
124    """
125    from meerschaum import _core
126    carrier, context = _carrier(self)
127    return int(_core.team_pipes_count(carrier, context, str(name), debug=debug))

Return how many pipes (and saved Pipelines) name a team (what renaming or deleting it changes).

def get_user_permissions(self, user: Any, debug: bool = False, **kwargs: Any) -> List[str]: 
130def get_user_permissions(self, user: Any, debug: bool = False, **kwargs: Any) -> List[str]:
131    """
132    Return a user's effective permissions on this instance.
133    """
134    from meerschaum import _core
135    carrier, context = _carrier(self)
136    return list(_core.get_user_permissions(carrier, context, _username(user), debug=debug))

Return a user's effective permissions on this instance.

def get_user_role(self, user: Any, debug: bool = False, **kwargs: Any) -> str: 
139def get_user_role(self, user: Any, debug: bool = False, **kwargs: Any) -> str:
140    """
141    Return a user's role on this instance (`'viewer'`, `'editor'`, `'operator'`, `'admin'` or
142    `'custom'`).
143    """
144    from meerschaum import _core
145    carrier, context = _carrier(self)
146    return str(_core.get_user_role(carrier, context, _username(user), debug=debug))

Return a user's role on this instance ('viewer', 'editor', 'operator', 'admin' or 'custom').

def set_user_attributes( self, user: Any, attributes: Dict[str, Any], debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
149def set_user_attributes(
150    self,
151    user: Any,
152    attributes: Dict[str, Any],
153    debug: bool = False,
154    **kwargs: Any
155) -> mrsm.SuccessTuple:
156    """
157    Replace a user's attributes (and nothing else).
158    """
159    from meerschaum import _core
160    carrier, context = _carrier(self)
161    return tuple(_core.set_user_attributes(
162        carrier, context, _username(user), dict(attributes or {}), debug=debug,
163    ))

Replace a user's attributes (and nothing else).

def lock_pipe( self, pipe: Pipe, locks: Any, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
166def lock_pipe(self, pipe: 'mrsm.Pipe', locks: Any, debug: bool = False, **kwargs: Any) -> mrsm.SuccessTuple:
167    """
168    Add locks to a registered pipe (`parameters['locks']`).
169    """
170    from meerschaum import _core
171    from meerschaum.core.Pipe._status import _status_context
172    return tuple(_core.pipe_lock(pipe, _status_context(pipe), list(locks), debug=debug))

Add locks to a registered pipe (parameters['locks']).

def unlock_pipe( self, pipe: Pipe, locks: Any, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
175def unlock_pipe(self, pipe: 'mrsm.Pipe', locks: Any, debug: bool = False, **kwargs: Any) -> mrsm.SuccessTuple:
176    """
177    Remove locks from a registered pipe (removing `'all'` removes every lock).
178    """
179    from meerschaum import _core
180    from meerschaum.core.Pipe._status import _status_context
181    return tuple(_core.pipe_unlock(pipe, _status_context(pipe), list(locks), debug=debug))

Remove locks from a registered pipe (removing 'all' removes every lock).

def set_pipe_access( self, pipe: Pipe, access: Dict[str, Any], debug: bool = False, **kwargs: Any) -> Tuple[bool, str]: 
184def set_pipe_access(
185    self,
186    pipe: 'mrsm.Pipe',
187    access: Dict[str, Any],
188    debug: bool = False,
189    **kwargs: Any
190) -> mrsm.SuccessTuple:
191    """
192    Store a registered pipe's access (`parameters['access']`).
193    """
194    from meerschaum import _core
195    from meerschaum.core.Pipe._status import _status_context
196    return tuple(_core.pipe_set_access(pipe, _status_context(pipe), dict(access), debug=debug))

Store a registered pipe's access (parameters['access']).

def make_connector(cls, _is_executor: bool = False): 
390def make_connector(cls, _is_executor: bool = False):
391    """
392    Register a class as a `Connector`.
393    The `type` will be the lower case of the class name, without the suffix `connector`.
394
395    Parameters
396    ----------
397    instance: bool, default False
398        If `True`, make this connector type an instance connector.
399        This requires implementing the various pipes functions and lots of testing.
400
401    Examples
402    --------
403    >>> import meerschaum as mrsm
404    >>> from meerschaum.connectors import make_connector, Connector
405    >>> 
406    >>> @make_connector
407    >>> class FooConnector(Connector):
408    ...     REQUIRED_ATTRIBUTES: list[str] = ['username', 'password']
409    ... 
410    >>> conn = mrsm.get_connector('foo:bar', username='dog', password='cat')
411    >>> print(conn.username, conn.password)
412    dog cat
413    >>> 
414    """
415    import re
416    from meerschaum.plugins import _get_parent_plugin
417    suffix_regex = (
418        r'connector$'
419        if not _is_executor
420        else r'executor$'
421    )
422    plugin_name = _get_parent_plugin(2)
423    typ = re.sub(suffix_regex, '', cls.__name__.lower())
424    with _locks['types']:
425        types[typ] = cls
426    with _locks['custom_types']:
427        custom_types.add(typ)
428    if plugin_name:
429        with _locks['plugins_types']:
430            if plugin_name not in plugins_types:
431                plugins_types[plugin_name] = []
432            plugins_types[plugin_name].append(typ)
433    with _locks['connectors']:
434        if typ not in connectors:
435            connectors[typ] = {}
436    if getattr(cls, 'IS_INSTANCE', False):
437        with _locks['instance_types']:
438            if typ not in instance_types:
439                instance_types.append(typ)
440
441    return cls

Register a class as a Connector. The type will be the lower case of the class name, without the suffix connector.

Parameters
  • instance (bool, default False): If True, make this connector type an instance connector. This requires implementing the various pipes functions and lots of testing.
Examples
>>> import meerschaum as mrsm
>>> from meerschaum.connectors import make_connector, Connector
>>> 
>>> @make_connector
>>> class FooConnector(Connector):
...     REQUIRED_ATTRIBUTES: list[str] = ['username', 'password']
... 
>>> conn = mrsm.get_connector('foo:bar', username='dog', password='cat')
>>> print(conn.username, conn.password)
dog cat
>>>
def entry( sysargs: List[str] | str | None = None, _patch_args: Dict[str, Any] | None = None, _use_cli_daemon: bool = True, _session_id: str | None = None) -> Tuple[bool, str]: 
114def entry(
115    sysargs: Union[List[str], str, None] = None,
116    _patch_args: Optional[Dict[str, Any]] = None,
117    _use_cli_daemon: bool = True,
118    _session_id: Optional[str] = None,
119) -> SuccessTuple:
120    """
121    Parse arguments and launch a Meerschaum action.
122
123    The action runs natively, in this process. Ctrl-C asks a looping action to stop and the call
124    then raises `KeyboardInterrupt`; Python delivers signals to the main thread only, so a call
125    made from another thread cannot be interrupted. Inside a plugin worker, a call made while a plugin module is being imported (at the top
126    level of the module) runs without plugins, so the import cannot recurse.
127
128    Returns
129    -------
130    A `SuccessTuple` indicating success.
131    """
132    return entry_without_daemon(sysargs, _patch_args=_patch_args)

Parse arguments and launch a Meerschaum action.

The action runs natively, in this process. Ctrl-C asks a looping action to stop and the call then raises KeyboardInterrupt; Python delivers signals to the main thread only, so a call made from another thread cannot be interrupted. Inside a plugin worker, a call made while a plugin module is being imported (at the top level of the module) runs without plugins, so the import cannot recurse.

Returns