meerschaum

Meerschaum banner

Meerschaum Python API

Welcome to the Meerschaum Python API technical documentation! Here you can find information about the classes and functions provided by the meerschaum package. Visit meerschaum.io for general usage documentation.

Root Module

For your convenience, the following classes and functions may be imported from the root meerschaum namespace:

Examples

Build a Connector

Get existing connectors or build a new one in-memory with the meerschaum.get_connector() factory function:

import meerschaum as mrsm

sql_conn = mrsm.get_connector(
    'sql:temp',
    flavor='sqlite',
    database='/tmp/tmp.db',
)
df = sql_conn.read("SELECT 1 AS foo")
print(df)
#    foo
# 0    1

sql_conn.to_sql(df, 'foo')
print(sql_conn.read('foo'))
#    foo
# 0    1

Create a Custom Connector Class

Decorate your connector classes with meerschaum.make_connector() to designate it as a custom connector:

from datetime import datetime, timezone
from random import randint
import meerschaum as mrsm
from meerschaum.utils.dtypes import round_time

@mrsm.make_connector
class FooConnector(mrsm.Connector):
    REQUIRED_ATTRIBUTES = ['username', 'password']

    def fetch(
        self,
        begin: datetime | None = None,
        end: datetime | None = None,
    ):
        now = begin or round_time(datetime.now(timezone.utc))
        return [
            {'ts': now, 'id': 1, 'vl': randint(1, 100)},
            {'ts': now, 'id': 2, 'vl': randint(1, 100)},
            {'ts': now, 'id': 3, 'vl': randint(1, 100)},
        ]

foo_conn = mrsm.get_connector(
    'foo:bar',
    username='foo',
    password='bar',
)
docs = foo_conn.fetch()

Build a Pipe

Build a meerschaum.Pipe in-memory:

from datetime import datetime
import meerschaum as mrsm

pipe = mrsm.Pipe(
    foo_conn, 'demo',
    instance=sql_conn,
    columns={'datetime': 'ts', 'id': 'id'},
    tags=['production'],
)
pipe.sync(begin=datetime(2024, 1, 1))
df = pipe.get_data()
print(df)
#           ts  id  vl
# 0 2024-01-01   1  97
# 1 2024-01-01   2  18
# 2 2024-01-01   3  96

Add temporary=True to skip registering the pipe in the pipes table.

Query an Integer-Axis Pipe by Datetime

When a pipe's datetime axis is an integer epoch, set precision so datetime bounds can be translated to the axis's integer value. This lets you pass a datetime begin / end to meerschaum.Pipe.get_data() (and to actions like show data, clear, and deduplicate):

from datetime import datetime, timezone
import meerschaum as mrsm

pipe = mrsm.Pipe(
    'demo', 'epoch',
    instance='sql:temp',
    columns={'datetime': 'ts'},
    dtypes={'ts': 'int'},
    precision='millisecond',
)
pipe.sync([{'ts': 1780099200000}])

### The datetime is translated to the epoch value `1780099200000`.
df = pipe.get_data(begin='2026-05-30')

Integer bounds (begin=1780099200000) still pass through unchanged. A datetime bound on a non-epoch integer axis (no precision set) raises a ValueError. Convert directly with meerschaum.utils.dtypes.datetime_to_int():

from datetime import datetime, timezone
from meerschaum.utils.dtypes import datetime_to_int

datetime_to_int(datetime(2026, 5, 30, tzinfo=timezone.utc), 'millisecond')
# 1780099200000

Get Registered Pipes

The meerschaum.get_pipes() function returns a dictionary hierarchy of pipes by connector, metric, and location:

import meerschaum as mrsm

pipes = mrsm.get_pipes(instance='sql:temp')
pipe = pipes['foo:bar']['demo'][None]

Add as_list=True to flatten the hierarchy:

import meerschaum as mrsm

pipes = mrsm.get_pipes(
    tags=['production'],
    instance=sql_conn,
    as_list=True,
)
print(pipes)
# [Pipe('foo:bar', 'demo', instance='sql:temp')]

Filter by the dtype of the datetime index column with datetime_dtypes. Accepted values are 'datetime', 'int', and 'None'; prefix with '_' to negate:

import meerschaum as mrsm

### Only pipes with a timestamp datetime index:
timestamp_pipes = mrsm.get_pipes(datetime_dtypes=['datetime'], as_list=True)

### Only pipes with an integer datetime index:
int_pipes = mrsm.get_pipes(datetime_dtypes=['int'], as_list=True)

### Exclude pipes without a datetime index:
datetime_pipes = mrsm.get_pipes(datetime_dtypes=['_None'], as_list=True)

Import Plugins

You can import a plugin's module through meerschaum.Plugin.module:

import meerschaum as mrsm

plugin = mrsm.Plugin('noaa')
with mrsm.Venv(plugin):
    noaa = plugin.module

If your plugin has submodules, use meerschaum.plugins.from_plugin_import:

from meerschaum.plugins import from_plugin_import
get_defined_pipes = from_plugin_import('compose.utils.pipes', 'get_defined_pipes')

Import multiple plugins with meerschaum.plugins.import_plugins:

from meerschaum.plugins import import_plugins
noaa, compose = import_plugins('noaa', 'compose')

Create a Job

Create a meerschaum.Job with name and sysargs:

import meerschaum as mrsm

job = mrsm.Job('syncing-engine', 'sync pipes --loop')
success, msg = job.start()

Pass executor_keys as the connectors keys of an API instance to create a remote job:

import meerschaum as mrsm

job = mrsm.Job(
    'foo',
    'sync pipes -s daily',
    executor_keys='api:main',
)

Import from a Virtual Environment Use the meerschaum.Venv context manager to activate a virtual environment:

import meerschaum as mrsm

with mrsm.Venv('noaa'):
    import requests

print(requests.__file__)
# /home/bmeares/.config/meerschaum/venvs/noaa/lib/python3.12/site-packages/requests/__init__.py

To import packages which may not be installed, use meerschaum.attempt_import():

import meerschaum as mrsm

requests = mrsm.attempt_import('requests', venv='noaa')
print(requests.__file__)
# /home/bmeares/.config/meerschaum/venvs/noaa/lib/python3.12/site-packages/requests/__init__.py

Run Actions

Run sysargs with meerschaum.entry():

import meerschaum as mrsm

success, msg = mrsm.entry('show pipes + show version : x2')

Use meerschaum.actions.get_action() to access an action function directly:

from meerschaum.actions import get_action

show_pipes = get_action(['show', 'pipes'])
success, msg = show_pipes(connector_keys=['plugin:noaa'])

Get a dictionary of available subactions with meerschaum.actions.get_subactions():

from meerschaum.actions import get_subactions

subactions = get_subactions('show')
success, msg = subactions['pipes']()

Create a Plugin

Run bootstrap plugin to create a new plugin:

mrsm bootstrap plugin example

This will create example.py in your plugins directory (default ~/.config/meerschaum/plugins/, Windows: %APPDATA%\Meerschaum\plugins). You may paste the example code from the "Create a Custom Action" example below.

Open your plugin with edit plugin:

mrsm edit plugin example

Run edit plugin and paste the example code below to try out the features.

See the writing plugins guide for more in-depth documentation.

Create a Custom Action

Decorate a function with meerschaum.actions.make_action to designate it as an action. Subactions will be automatically detected if not decorated:

from meerschaum.actions import make_action

@make_action
def sing():
    print('What would you like me to sing?')
    return True, "Success"

def sing_tune():
    return False, "I don't know that song!"

def sing_song():
    print('Hello, World!')
    return True, "Success"

Use meerschaum.plugins.add_plugin_argument() to create new parameters for your action:

from meerschaum.plugins import make_action, add_plugin_argument

add_plugin_argument(
    '--song', type=str, help='What song to sing.',
)

@make_action
def sing_melody(action=None, song=None):
    to_sing = action[0] if action else song
    if not to_sing:
        return False, "Please tell me what to sing!"

    return True, f'~I am singing {to_sing}~'
mrsm sing melody lalala

mrsm sing melody --song do-re-mi

Add a Page to the Web Dashboard Use the decorators meerschaum.plugins.dash_plugin() and meerschaum.plugins.web_page() to add new pages to the web dashboard:

from meerschaum.plugins import dash_plugin, web_page

@dash_plugin
def init_dash(dash_app):

    import dash.html as html
    import dash_bootstrap_components as dbc
    from dash import Input, Output, no_update

    ### Routes to '/dash/my-page'
    @web_page('/my-page', login_required=False)
    def my_page():
        return dbc.Container([
            html.H1("Hello, World!"),
            dbc.Button("Click me", id='my-button'),
            html.Div(id="my-output-div"),
        ])

    @dash_app.callback(
        Output('my-output-div', 'children'),
        Input('my-button', 'n_clicks'),
    )
    def my_button_click(n_clicks):
        if not n_clicks:
            return no_update
        return html.P(f'You clicked {n_clicks} times!')

Submodules

meerschaum.actions
Access functions for actions and subactions.

meerschaum.config
Read and write the Meerschaum configuration registry.

meerschaum.connectors
Build connectors to interact with databases and fetch data.

meerschaum.jobs
Start background jobs.

meerschaum.plugins
Access plugin modules and other API utilties.

meerschaum.utils
Utility functions are available in several submodules:

 1#! /usr/bin/env python
 2# -*- coding: utf-8 -*-
 3# vim:fenc=utf-8
 4
 5"""
 6Copyright 2020–2026 Bennett Meares
 7
 8Licensed under the Apache License, Version 2.0 (the "License");
 9you may not use this file except in compliance with the License.
10You may obtain a copy of the License at
11
12   http://www.apache.org/licenses/LICENSE-2.0
13
14Unless required by applicable law or agreed to in writing, software
15distributed under the License is distributed on an "AS IS" BASIS,
16WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17See the License for the specific language governing permissions and
18limitations under the License.
19"""
20
21import atexit
22
23from meerschaum.utils.typing import SuccessTuple
24from meerschaum.utils.packages import attempt_import
25from meerschaum.core.Pipe import Pipe
26from meerschaum.plugins import Plugin
27from meerschaum.utils.venv import Venv
28from meerschaum.jobs import Job, make_executor
29from meerschaum.connectors import get_connector, Connector, InstanceConnector, make_connector
30from meerschaum.utils import get_pipes
31from meerschaum.utils.formatting import pprint
32from meerschaum._internal.docs import index as __doc__
33from meerschaum.config import __version__, get_config
34from meerschaum._internal.entry import entry
35from meerschaum.__main__ import _close_pools
36
37atexit.register(_close_pools)
38
39__pdoc__ = {'gui': False, 'api': False, 'core': False, '_internal': False}
40__all__ = (
41    "get_pipes",
42    "get_connector",
43    "get_config",
44    "Pipe",
45    "Plugin",
46    "SuccessTuple",
47    "Venv",
48    "Plugin",
49    "Job",
50    "pprint",
51    "attempt_import",
52    "actions",
53    "config",
54    "connectors",
55    "jobs",
56    "plugins",
57    "utils",
58    "SuccessTuple",
59    "Connector",
60    "InstanceConnector",
61    "make_connector",
62    "entry",
63)
def get_pipes( connector_keys: Union[str, List[str], NoneType] = None, metric_keys: Union[str, List[str], NoneType] = None, location_keys: Union[str, List[str], NoneType] = None, tags: Optional[List[str]] = None, targets: Optional[List[str]] = None, datetime_dtypes: Optional[List[str]] = None, params: Optional[Dict[str, Any]] = None, mrsm_instance: Union[str, InstanceConnector, NoneType] = None, instance: Union[str, InstanceConnector, NoneType] = None, as_list: bool = False, as_tags_dict: bool = False, as_targets_dict: bool = False, method: str = 'registered', workers: Optional[int] = None, debug: bool = False, _cache_parameters: bool = True, **kw: Any) -> Union[Dict[str, Dict[str, Dict[Optional[str], Pipe]]], List[Pipe], Dict[str, Pipe]]:
 29def get_pipes(
 30    connector_keys: Union[str, List[str], None] = None,
 31    metric_keys: Union[str, List[str], None] = None,
 32    location_keys: Union[str, List[str], None] = None,
 33    tags: Optional[List[str]] = None,
 34    targets: Optional[List[str]] = None,
 35    datetime_dtypes: Optional[List[str]] = None,
 36    params: Optional[Dict[str, Any]] = None,
 37    mrsm_instance: Union[str, InstanceConnector, None] = None,
 38    instance: Union[str, InstanceConnector, None] = None,
 39    as_list: bool = False,
 40    as_tags_dict: bool = False,
 41    as_targets_dict: bool = False,
 42    method: str = 'registered',
 43    workers: Optional[int] = None,
 44    debug: bool = False,
 45    _cache_parameters: bool = True,
 46    **kw: Any
 47) -> Union[PipesDict, List[mrsm.Pipe], Dict[str, mrsm.Pipe]]:
 48    """
 49    Return a dictionary or list of `meerschaum.Pipe` objects.
 50
 51    Parameters
 52    ----------
 53    connector_keys: Union[str, List[str], None], default None
 54        String or list of connector keys.
 55        If omitted or is `'*'`, fetch all possible keys.
 56        If a string begins with `'_'`, select keys that do NOT match the string.
 57
 58    metric_keys: Union[str, List[str], None], default None
 59        String or list of metric keys. See `connector_keys` for formatting.
 60
 61    location_keys: Union[str, List[str], None], default None
 62        String or list of location keys. See `connector_keys` for formatting.
 63
 64    tags: Optional[List[str]], default None
 65        If provided, only include pipes with these tags.
 66
 67    datetime_dtypes: Optional[List[str]], default None
 68        If provided, only include pipes with the corresponding `datetime` axis dtypes.
 69        Accepted values are `datetime`, `int`, `None` (or `null`, etc.).
 70        May be negated by `_`.
 71
 72    params: Optional[Dict[str, Any]], default None
 73        Dictionary of additional parameters to search by.
 74        Params are parsed into a SQL WHERE clause.
 75        E.g. `{'a': 1, 'b': 2}` equates to `'WHERE a = 1 AND b = 2'`
 76
 77    mrsm_instance: Union[str, InstanceConnector, None], default None
 78        Connector keys for the Meerschaum instance of the pipes.
 79        Must be a `meerschaum.connectors.sql.SQLConnector.SQLConnector` or
 80        `meerschaum.connectors.api.APIConnector.APIConnector`.
 81        
 82    as_list: bool, default False
 83        If `True`, return pipes in a list instead of a hierarchical dictionary.
 84        `False` : `{connector_keys: {metric_key: {location_key: Pipe}}}`
 85        `True`  : `[Pipe]`
 86
 87    as_tags_dict: bool, default False
 88        If `True`, return a dictionary mapping tags to pipes.
 89        Pipes with multiple tags will be repeated.
 90
 91    as_targets_dict: bool, default False
 92        If `True`, return a dictionary mapping `(schema, target)` tuples to pipes.
 93        Pipes sharing the same target across different schemata are grouped separately.
 94
 95    method: str, default 'registered'
 96        Available options: `['registered', 'explicit', 'all']`
 97        If `'registered'` (default), create pipes based on registered keys in the connector's pipes table
 98        (API or SQL connector, depends on mrsm_instance).
 99        If `'explicit'`, create pipes from provided connector_keys, metric_keys, and location_keys
100        instead of consulting the pipes table. Useful for creating non-existent pipes.
101        If `'all'`, create pipes from predefined metrics and locations. Required `connector_keys`.
102        **NOTE:** Method `'all'` is not implemented!
103
104    workers: Optional[int], default None
105        If provided (and `as_tags_dict` or `as_targets_dict` is `True`), set the number of workers
106        for the pool to fetch tags or targets.
107        Only takes effect if the instance connector supports multi-threading.
108
109    **kw: Any:
110        Keyword arguments to pass to the `meerschaum.Pipe` constructor.
111
112    Returns
113    -------
114    A dictionary of dictionaries and `meerschaum.Pipe` objects
115    in the connector, metric, location hierarchy.
116    If `as_list` is `True`, return a list of `meerschaum.Pipe` objects.
117    If `as_tags_dict` is `True`, return a dictionary mapping tags to pipes.
118    If `as_targets_dict` is `True`, return a dictionary mapping targets to pipes.
119
120    Examples
121    --------
122    ```
123    >>> ### Manual definition:
124    >>> pipes = {
125    ...     <connector_keys>: {
126    ...         <metric_key>: {
127    ...             <location_key>: Pipe(
128    ...                 <connector_keys>,
129    ...                 <metric_key>,
130    ...                 <location_key>,
131    ...             ),
132    ...         },
133    ...     },
134    ... },
135    >>> ### Accessing a single pipe:
136    >>> pipes['sql:main']['weather'][None]
137    >>> ### Return a list instead:
138    >>> get_pipes(as_list=True)
139    [Pipe('sql:main', 'weather')]
140    >>> get_pipes(as_tags_dict=True)
141    {'gvl': Pipe('sql:main', 'weather')}
142    ```
143    """
144    import json
145    from collections import defaultdict
146    from meerschaum.config import get_config
147    from meerschaum.config.static import STATIC_CONFIG
148    from meerschaum.utils.warnings import error
149    from meerschaum.utils.misc import filter_keywords, separate_negation_values
150    from meerschaum.utils.pool import get_pool
151    from meerschaum.utils.pipes import replace_pipes_syntax
152    from meerschaum.utils.debug import dprint
153    from meerschaum.utils.dtypes import value_is_null, get_current_timestamp
154    from meerschaum import Pipe
155
156    negation_prefix = STATIC_CONFIG['system']['fetch_pipes_keys']['negation_prefix']
157    if datetime_dtypes:
158        if isinstance(datetime_dtypes, str):
159            datetime_dtypes = [datetime_dtypes]
160        for _dt in datetime_dtypes:
161            _clean = str(_dt).lstrip(negation_prefix).lower()
162            if _clean not in ('datetime', 'int') and not value_is_null(_clean):
163                error(f"Invalid datetime dtype '{_dt}'.")
164
165    if connector_keys is None:
166        connector_keys = []
167    if metric_keys is None:
168        metric_keys = []
169    if location_keys is None:
170        location_keys = []
171    if params is None:
172        params = {}
173    if tags is None:
174        tags = []
175
176    if isinstance(connector_keys, str):
177        connector_keys = [connector_keys]
178    if isinstance(metric_keys, str):
179        metric_keys = [metric_keys]
180    if isinstance(location_keys, str):
181        location_keys = [location_keys]
182
183    ### Get SQL or API connector (keys come from `connector.fetch_pipes_keys()`).
184    if mrsm_instance is None:
185        mrsm_instance = instance
186    if mrsm_instance is None:
187        mrsm_instance = get_config('meerschaum', 'instance', patch=True)
188    if isinstance(mrsm_instance, str):
189        from meerschaum.connectors.parse import parse_instance_keys
190        connector = parse_instance_keys(keys=mrsm_instance, debug=debug)
191    else:
192        from meerschaum.connectors import instance_types
193        valid_connector = False
194        if hasattr(mrsm_instance, 'type'):
195            if mrsm_instance.type in instance_types:
196                valid_connector = True
197        if not valid_connector:
198            error(f"Invalid instance connector: {mrsm_instance}")
199        connector = mrsm_instance
200    if debug:
201        dprint(f"Using instance connector: {connector}")
202    if not connector:
203        error(f"Could not create connector from keys: '{mrsm_instance}'")
204
205    ### Get a list of tuples for the keys needed to build pipes.
206    result = fetch_pipes_keys(
207        method,
208        connector,
209        connector_keys = connector_keys,
210        metric_keys = metric_keys,
211        location_keys = location_keys,
212        tags = tags,
213        params = params,
214        workers = workers,
215        debug = debug
216    )
217    if result is None:
218        error("Unable to build pipes!")
219    result_items: List[Tuple] = (
220        list(result.items())
221        if isinstance(result, dict)
222        else [(None, keys_tuple) for keys_tuple in result]
223    )
224
225    ### Populate the `pipes` dictionary with Pipes based on the keys
226    ### obtained from the chosen `method`.
227    in_dtypes, ex_dtypes = separate_negation_values(datetime_dtypes or [])
228    in_targets, ex_targets = separate_negation_values(targets or [])
229    pipes: PipesDict = {}
230    targets_pipes: Dict[Tuple[Optional[str], str], List[mrsm.Pipe]] = defaultdict(lambda: [])
231    connector_schema = getattr(connector, 'schema', None)
232    connector_is_sql = getattr(connector, 'type', None) == 'sql'
233    connector_flavor = getattr(connector, 'flavor', None)
234    for pipe_id, keys_tuple in result_items:
235        ck, mk, lk = keys_tuple[0], keys_tuple[1], keys_tuple[2]
236        pipe_tags_or_parameters = keys_tuple[3] if len(keys_tuple) == 4 else None
237        pipe_parameters = (
238            pipe_tags_or_parameters
239            if isinstance(pipe_tags_or_parameters, (dict, str))
240            else None
241        )
242        if isinstance(pipe_parameters, str):
243            pipe_parameters = json.loads(pipe_parameters)
244        pipe_tags = (
245            pipe_tags_or_parameters
246            if isinstance(pipe_tags_or_parameters, list)
247            else (
248                pipe_tags_or_parameters.get('tags', [])
249                if isinstance(pipe_tags_or_parameters, dict)
250                else None
251            )
252        )
253
254        pipe = Pipe(
255            ck, mk, lk,
256            mrsm_instance = connector,
257            parameters = pipe_parameters,
258            tags = pipe_tags,
259            debug = debug,
260            **filter_keywords(Pipe, **kw)
261        )
262        pipe.__dict__['_tags'] = pipe_tags
263        if pipe_id is not None:
264            pipe._cache_value('_id', pipe_id, memory_only=True, debug=debug)
265        if pipe_parameters is not None:
266            now = get_current_timestamp('ms', as_int=True) / 1000
267            full_attributes = {
268                'connector_keys': ck,
269                'metric_key': mk,
270                'location_key': lk,
271                'parameters': pipe_parameters,
272            }
273            if pipe_id is not None:
274                full_attributes['pipe_id'] = pipe_id
275            pipe._cache_value('attributes', full_attributes, memory_only=True, debug=debug)
276            pipe._cache_value('_attributes_sync_time', now, memory_only=True, debug=debug)
277
278        if datetime_dtypes or targets:
279            parameters_str = str(pipe_parameters)
280            if pipe_parameters is None or 'MRSM{' in parameters_str or 'Pipe(' in parameters_str:
281                pipe_parameters = pipe.get_parameters(debug=debug)
282
283        keep_pipe = True
284
285        if datetime_dtypes:
286            columns_val = (pipe_parameters or {}).get('columns', {}) or {}
287            dt_col = columns_val.get('datetime', None)
288            pipe_dtypes = (
289                ((pipe_parameters or {}).get('dtypes', None) or {})
290                if dt_col
291                else None
292            )
293            dt_typ = pipe_dtypes.get(dt_col, None) if dt_col else None
294
295            def _dtype_matches(clean_d):
296                if not dt_col:
297                    return value_is_null(clean_d)
298                return (
299                    (clean_d == 'int' and 'int' in str(dt_typ).lower())
300                    or
301                    (clean_d == 'datetime' and 'int' not in str(dt_typ).lower())
302                )
303
304            in_match = not in_dtypes or any(_dtype_matches(d) for d in in_dtypes)
305            ex_match = bool(ex_dtypes and any(_dtype_matches(d) for d in ex_dtypes))
306            keep_pipe = keep_pipe and in_match and not ex_match
307            if not keep_pipe:
308                continue
309
310        if targets:
311            pipe_target = pipe.target
312            in_target_match = not in_targets or any(t == pipe_target for t in in_targets)
313            ex_target_match = bool(ex_targets and any(t == pipe_target for t in ex_targets))
314            keep_pipe = keep_pipe and in_target_match and not ex_target_match
315            if not keep_pipe:
316                continue
317
318        if ck not in pipes:
319            pipes[ck] = {}
320
321        if mk not in pipes[ck]:
322            pipes[ck][mk] = {}
323
324
325        pipes[ck][mk][lk] = pipe
326
327        if as_targets_dict:
328            raw_params = pipe_parameters if isinstance(pipe_parameters, dict) else {}
329            schema = raw_params.get('schema') or connector_schema
330            explicit_target = (
331                raw_params.get('target')
332                or raw_params.get('target_name')
333                or raw_params.get('target_table')
334                or raw_params.get('target_table_name')
335            )
336            if explicit_target:
337                target_name = (
338                    replace_pipes_syntax(explicit_target, _pipe=pipe)
339                    if isinstance(explicit_target, str) and '{{' in explicit_target
340                    else explicit_target
341                )
342            else:
343                target_name = pipe._target_legacy()
344                if connector_is_sql and connector_flavor:
345                    from meerschaum.utils.sql import truncate_item_name
346                    target_name = truncate_item_name(target_name, connector_flavor)
347            targets_pipes[(schema, target_name)].append(pipe)
348
349    if not as_list and not as_tags_dict and not as_targets_dict:
350        return pipes
351
352    from meerschaum.utils.pipes import flatten_pipes_dict
353    pipes_list = flatten_pipes_dict(pipes)
354    if as_list:
355        return pipes_list
356
357    pool = get_pool(workers=(workers if connector.IS_THREAD_SAFE else 1))
358
359    def gather_pipe_tags(pipe: mrsm.Pipe) -> Tuple[mrsm.Pipe, List[str]]:
360        _tags = pipe.__dict__.get('_tags', None)
361        gathered_tags = _tags if _tags is not None else pipe.tags
362        return pipe, (gathered_tags or [])
363
364    if as_tags_dict:
365        tags_pipes = defaultdict(lambda: [])
366        pipes_tags = dict(pool.map(gather_pipe_tags, pipes_list))
367        for pipe, tags in pipes_tags.items():
368            for tag in (tags or []):
369                tags_pipes[tag].append(pipe)
370
371        return dict(tags_pipes)
372
373    if as_targets_dict:
374        return dict(targets_pipes)
375
376    raise NotImplementedError("No futher options for returning pipes.")

Return a dictionary or list of meerschaum.Pipe objects.

Parameters
  • connector_keys (Union[str, List[str], None], default None): String or list of connector keys. If omitted or is '*', fetch all possible keys. If a string begins with '_', select keys that do NOT match the string.
  • metric_keys (Union[str, List[str], None], default None): String or list of metric keys. See connector_keys for formatting.
  • location_keys (Union[str, List[str], None], default None): String or list of location keys. See connector_keys for formatting.
  • tags (Optional[List[str]], default None): If provided, only include pipes with these tags.
  • datetime_dtypes (Optional[List[str]], default None): If provided, only include pipes with the corresponding datetime axis dtypes. Accepted values are datetime, int, None (or null, etc.). May be negated by _.
  • params (Optional[Dict[str, Any]], default None): Dictionary of additional parameters to search by. Params are parsed into a SQL WHERE clause. E.g. {'a': 1, 'b': 2} equates to 'WHERE a = 1 AND b = 2'
  • mrsm_instance (Union[str, InstanceConnector, None], default None): Connector keys for the Meerschaum instance of the pipes. Must be a meerschaum.connectors.sql.SQLConnector.SQLConnector or meerschaum.connectors.api.APIConnector.APIConnector.
  • as_list (bool, default False): If True, return pipes in a list instead of a hierarchical dictionary. False : {connector_keys: {metric_key: {location_key: Pipe}}} True : [Pipe]
  • as_tags_dict (bool, default False): If True, return a dictionary mapping tags to pipes. Pipes with multiple tags will be repeated.
  • as_targets_dict (bool, default False): If True, return a dictionary mapping (schema, target) tuples to pipes. Pipes sharing the same target across different schemata are grouped separately.
  • method (str, default 'registered'): Available options: ['registered', 'explicit', 'all'] If 'registered' (default), create pipes based on registered keys in the connector's pipes table (API or SQL connector, depends on mrsm_instance). If 'explicit', create pipes from provided connector_keys, metric_keys, and location_keys instead of consulting the pipes table. Useful for creating non-existent pipes. If 'all', create pipes from predefined metrics and locations. Required connector_keys. NOTE: Method 'all' is not implemented!
  • workers (Optional[int], default None): If provided (and as_tags_dict or as_targets_dict is True), set the number of workers for the pool to fetch tags or targets. Only takes effect if the instance connector supports multi-threading.
  • **kw (Any:): Keyword arguments to pass to the meerschaum.Pipe constructor.
Returns
  • A dictionary of dictionaries and meerschaum.Pipe objects
  • in the connector, metric, location hierarchy.
  • If as_list is True, return a list of meerschaum.Pipe objects.
  • If as_tags_dict is True, return a dictionary mapping tags to pipes.
  • If as_targets_dict is True, return a dictionary mapping targets to pipes.
Examples
>>> ### Manual definition:
>>> pipes = {
...     <connector_keys>: {
...         <metric_key>: {
...             <location_key>: Pipe(
...                 <connector_keys>,
...                 <metric_key>,
...                 <location_key>,
...             ),
...         },
...     },
... },
>>> ### Accessing a single pipe:
>>> pipes['sql:main']['weather'][None]
>>> ### Return a list instead:
>>> get_pipes(as_list=True)
[Pipe('sql:main', 'weather')]
>>> get_pipes(as_tags_dict=True)
{'gvl': Pipe('sql:main', 'weather')}
def get_connector( type: str = None, label: str = None, refresh: bool = False, debug: bool = False, _load_plugins: bool = True, **kw: Any) -> Connector:
 68def get_connector(
 69    type: str = None,
 70    label: str = None,
 71    refresh: bool = False,
 72    debug: bool = False,
 73    _load_plugins: bool = True,
 74    **kw: Any
 75) -> Connector:
 76    """
 77    Return existing connector or create new connection and store for reuse.
 78    
 79    You can create new connectors if enough parameters are provided for the given type and flavor.
 80
 81    Parameters
 82    ----------
 83    type: Optional[str], default None
 84        Connector type (sql, api, etc.).
 85        Defaults to the type of the configured `instance_connector`.
 86
 87    label: Optional[str], default None
 88        Connector label (e.g. main). Defaults to `'main'`.
 89
 90    refresh: bool, default False
 91        Refresh the Connector instance / construct new object. Defaults to `False`.
 92
 93    kw: Any
 94        Other arguments to pass to the Connector constructor.
 95        If the Connector has already been constructed and new arguments are provided,
 96        `refresh` is set to `True` and the old Connector is replaced.
 97
 98    Returns
 99    -------
100    A new Meerschaum connector (e.g. `meerschaum.connectors.api.APIConnector`,
101    `meerschaum.connectors.sql.SQLConnector`).
102    
103    Examples
104    --------
105    The following parameters would create a new
106    `meerschaum.connectors.sql.SQLConnector` that isn't in the configuration file.
107
108    ```
109    >>> conn = get_connector(
110    ...     type = 'sql',
111    ...     label = 'newlabel',
112    ...     flavor = 'sqlite',
113    ...     database = '/file/path/to/database.db'
114    ... )
115    >>>
116    ```
117
118    """
119    from meerschaum.connectors.parse import parse_instance_keys
120    from meerschaum.config import get_config
121    from meerschaum._internal.static import STATIC_CONFIG
122    from meerschaum.utils.warnings import warn
123    global _loaded_plugin_connectors
124    if isinstance(type, str) and not label and ':' in type:
125        type, label = type.split(':', maxsplit=1)
126
127    if _load_plugins:
128        with _locks['_loaded_plugin_connectors']:
129            if not _loaded_plugin_connectors:
130                load_plugin_connectors()
131                _load_builtin_custom_connectors()
132                _loaded_plugin_connectors = True
133
134    if type is None and label is None:
135        default_instance_keys = get_config('meerschaum', 'instance', patch=True)
136        ### recursive call to get_connector
137        return parse_instance_keys(default_instance_keys)
138
139    ### NOTE: the default instance connector may not be main.
140    ### Only fall back to 'main' if the type is provided by the label is omitted.
141    label = label if label is not None else STATIC_CONFIG['connectors']['default_label']
142
143    ### type might actually be a label. Check if so and raise a warning.
144    if type not in connectors:
145        possibilities, poss_msg = [], ""
146        for _type in get_config('meerschaum', 'connectors'):
147            if type in get_config('meerschaum', 'connectors', _type):
148                possibilities.append(f"{_type}:{type}")
149        if len(possibilities) > 0:
150            poss_msg = " Did you mean"
151            for poss in possibilities[:-1]:
152                poss_msg += f" '{poss}',"
153            if poss_msg.endswith(','):
154                poss_msg = poss_msg[:-1]
155            if len(possibilities) > 1:
156                poss_msg += " or"
157            poss_msg += f" '{possibilities[-1]}'?"
158
159        warn(f"Cannot create Connector of type '{type}'." + poss_msg, stack=False)
160        return None
161
162    if 'sql' not in types:
163        from meerschaum.connectors.plugin import PluginConnector
164        from meerschaum.connectors.valkey import ValkeyConnector
165        with _locks['types']:
166            types.update({
167                'api': APIConnector,
168                'sql': SQLConnector,
169                'plugin': PluginConnector,
170                'valkey': ValkeyConnector,
171            })
172
173    ### determine if we need to call the constructor
174    if not refresh:
175        ### see if any user-supplied arguments differ from the existing instance
176        if label in connectors[type]:
177            warning_message = None
178            for attribute, value in kw.items():
179                if attribute not in connectors[type][label].meta:
180                    import inspect
181                    cls = connectors[type][label].__class__
182                    cls_init_signature = inspect.signature(cls)
183                    cls_init_params = cls_init_signature.parameters
184                    if attribute not in cls_init_params:
185                        warning_message = (
186                            f"Received new attribute '{attribute}' not present in connector " +
187                            f"{connectors[type][label]}.\n"
188                        )
189                elif connectors[type][label].__dict__[attribute] != value:
190                    warning_message = (
191                        f"Mismatched values for attribute '{attribute}' in connector "
192                        + f"'{connectors[type][label]}'.\n" +
193                        f"  - Keyword value: '{value}'\n" +
194                        f"  - Existing value: '{connectors[type][label].__dict__[attribute]}'\n"
195                    )
196            if warning_message is not None:
197                warning_message += (
198                    "\nSetting `refresh` to True and recreating connector with type:"
199                    + f" '{type}' and label '{label}'."
200                )
201                refresh = True
202                warn(warning_message)
203        else: ### connector doesn't yet exist
204            refresh = True
205
206    ### only create an object if refresh is True
207    ### (can be manually specified, otherwise determined above)
208    if refresh:
209        with _locks['connectors']:
210            try:
211                ### will raise an error if configuration is incorrect / missing
212                conn = types[type](label=label, **kw)
213                connectors[type][label] = conn
214            except InvalidAttributesError as ie:
215                warn(
216                    f"Incorrect attributes for connector '{type}:{label}'.\n"
217                    + str(ie),
218                    stack = False,
219                )
220                conn = None
221            except Exception as e:
222                from meerschaum.utils.formatting import get_console
223                console = get_console()
224                if console:
225                    console.print_exception()
226                warn(
227                    f"Exception when creating connector '{type}:{label}'.\n" + str(e),
228                    stack = False,
229                )
230                conn = None
231        if conn is None:
232            return None
233
234    return connectors[type][label]

Return existing connector or create new connection and store for reuse.

You can create new connectors if enough parameters are provided for the given type and flavor.

Parameters
  • type (Optional[str], default None): Connector type (sql, api, etc.). Defaults to the type of the configured instance_connector.
  • label (Optional[str], default None): Connector label (e.g. main). Defaults to 'main'.
  • refresh (bool, default False): Refresh the Connector instance / construct new object. Defaults to False.
  • kw (Any): Other arguments to pass to the Connector constructor. If the Connector has already been constructed and new arguments are provided, refresh is set to True and the old Connector is replaced.
Returns
Examples

The following parameters would create a new meerschaum.connectors.sql.SQLConnector that isn't in the configuration file.

>>> conn = get_connector(
...     type = 'sql',
...     label = 'newlabel',
...     flavor = 'sqlite',
...     database = '/file/path/to/database.db'
... )
>>>
def get_config( *keys: str, patch: bool = True, substitute: bool = True, sync_files: bool = True, write_missing: bool = True, as_tuple: bool = False, warn: bool = True, debug: bool = False) -> Any:
112def get_config(
113    *keys: str,
114    patch: bool = True,
115    substitute: bool = True,
116    sync_files: bool = True,
117    write_missing: bool = True,
118    as_tuple: bool = False,
119    warn: bool = True,
120    debug: bool = False
121) -> Any:
122    """
123    Return the Meerschaum configuration dictionary.
124    If positional arguments are provided, index by the keys.
125    Raises a warning if invalid keys are provided.
126
127    Parameters
128    ----------
129    keys: str:
130        List of strings to index.
131
132    patch: bool, default True
133        If `True`, patch missing default keys into the config directory.
134        Defaults to `True`.
135
136    sync_files: bool, default True
137        If `True`, sync files if needed.
138        Defaults to `True`.
139
140    write_missing: bool, default True
141        If `True`, write default values when the main config files are missing.
142        Defaults to `True`.
143
144    substitute: bool, default True
145        If `True`, subsitute 'MRSM{}' values.
146        Defaults to `True`.
147
148    as_tuple: bool, default False
149        If `True`, return a tuple of type (success, value).
150        Defaults to `False`.
151        
152    Returns
153    -------
154    The value in the configuration directory, indexed by the provided keys.
155
156    Examples
157    --------
158    >>> get_config('meerschaum', 'instance')
159    'sql:main'
160    >>> get_config('does', 'not', 'exist')
161    UserWarning: Invalid keys in config: ('does', 'not', 'exist')
162    """
163    import json
164
165    symlinks_key = STATIC_CONFIG['config']['symlinks_key']
166    if debug:
167        from meerschaum.utils.debug import dprint
168        dprint(f"Indexing keys: {keys}", color=False)
169
170    if len(keys) == 0:
171        _rc = _config(
172            substitute=substitute,
173            sync_files=sync_files,
174            write_missing=(write_missing and _allow_write_missing),
175        )
176        if as_tuple:
177            return True, _rc 
178        return _rc
179    
180    ### Weird threading issues, only import if substitute is True.
181    if substitute:
182        from meerschaum.config._read_config import search_and_substitute_config
183    ### Invalidate the cache if it was read before with substitute=False
184    ### but there still exist substitutions.
185    if (
186        config is not None and substitute and keys[0] != symlinks_key
187        and 'MRSM{' in json.dumps(config.get(keys[0]))
188    ):
189        try:
190            _subbed = search_and_substitute_config({keys[0]: config[keys[0]]})
191        except Exception:
192            import traceback
193            traceback.print_exc()
194            _subbed = {keys[0]: config[keys[0]]}
195
196        config[keys[0]] = _subbed[keys[0]]
197        if symlinks_key in _subbed:
198            if symlinks_key not in config:
199                config[symlinks_key] = {}
200            config[symlinks_key] = apply_patch_to_config(
201                _subbed.get(symlinks_key, {}),
202                config.get(symlinks_key, {}),
203            )
204
205    from meerschaum.config._sync import sync_files as _sync_files
206    if config is None:
207        _config(*keys, sync_files=sync_files)
208
209    invalid_keys = False
210    if keys[0] not in config and keys[0] != symlinks_key:
211        single_key_config = read_config(
212            keys=[keys[0]], substitute=substitute, write_missing=write_missing
213        )
214        if keys[0] not in single_key_config:
215            invalid_keys = True
216        else:
217            config[keys[0]] = single_key_config.get(keys[0], None)
218            if symlinks_key in single_key_config and keys[0] in single_key_config[symlinks_key]:
219                if symlinks_key not in config:
220                    config[symlinks_key] = {}
221                config[symlinks_key][keys[0]] = single_key_config[symlinks_key][keys[0]]
222
223            if sync_files:
224                _sync_files(keys=[keys[0]])
225
226    c = config
227    if len(keys) > 0:
228        for k in keys:
229            try:
230                c = c[k]
231            except Exception:
232                invalid_keys = True
233                break
234        if invalid_keys:
235            ### Check if the keys are in the default configuration.
236            from meerschaum.config._default import default_config
237            in_default = True
238            patched_default_config = (
239                search_and_substitute_config(default_config)
240                if substitute else copy.deepcopy(default_config)
241            )
242            _c = patched_default_config
243            for k in keys:
244                try:
245                    _c = _c[k]
246                except Exception:
247                    in_default = False
248            if in_default:
249                c = _c
250                invalid_keys = False
251            warning_msg = f"Invalid keys in config: {keys}"
252            if not in_default:
253                try:
254                    if warn:
255                        from meerschaum.utils.warnings import warn as _warn
256                        _warn(warning_msg, stacklevel=3, color=False)
257                except Exception:
258                    if warn:
259                        print(warning_msg)
260                if as_tuple:
261                    return False, None
262                return None
263
264            ### Don't write keys that we haven't yet loaded into memory.
265            not_loaded_keys = [k for k in patched_default_config if k not in config]
266            for k in not_loaded_keys:
267                patched_default_config.pop(k, None)
268
269            set_config(
270                apply_patch_to_config(
271                    patched_default_config,
272                    config,
273                )
274            )
275            if patch and keys[0] != symlinks_key and write_missing:
276                ### Only persist defaults when the key's file is genuinely absent.
277                ### Never overwrite an existing file (e.g. one that failed to parse) ─
278                ### doing so would clobber the user's config with default values.
279                ### Brand-new config files are still created by `read_config()`.
280                from meerschaum.config._read_config import get_keyfile_path
281                keyfile_exists = get_keyfile_path(keys[0], create_new=False) is not None
282                if not keyfile_exists:
283                    write_config(config, debug=debug)
284
285    if as_tuple:
286        return (not invalid_keys), c
287    return c

Return the Meerschaum configuration dictionary. If positional arguments are provided, index by the keys. Raises a warning if invalid keys are provided.

Parameters
  • keys (str:): List of strings to index.
  • patch (bool, default True): If True, patch missing default keys into the config directory. Defaults to True.
  • sync_files (bool, default True): If True, sync files if needed. Defaults to True.
  • write_missing (bool, default True): If True, write default values when the main config files are missing. Defaults to True.
  • substitute (bool, default True): If True, subsitute 'MRSM{}' values. Defaults to True.
  • as_tuple (bool, default False): If True, return a tuple of type (success, value). Defaults to False.
Returns
  • The value in the configuration directory, indexed by the provided keys.
Examples
>>> get_config('meerschaum', 'instance')
'sql:main'
>>> get_config('does', 'not', 'exist')
UserWarning: Invalid keys in config: ('does', 'not', 'exist')
class Pipe:
 66class Pipe:
 67    """
 68    Access Meerschaum pipes via Pipe objects.
 69    
 70    Pipes are identified by the following:
 71
 72    1. Connector keys (e.g. `'sql:main'`)
 73    2. Metric key (e.g. `'weather'`)
 74    3. Location (optional; e.g. `None`)
 75    
 76    A pipe's connector keys correspond to a data source, and when the pipe is synced,
 77    its `fetch` definition is evaluated and executed to produce new data.
 78    
 79    Alternatively, new data may be directly synced via `pipe.sync()`:
 80    
 81    ```
 82    >>> from meerschaum import Pipe
 83    >>> pipe = Pipe('csv', 'weather')
 84    >>>
 85    >>> import pandas as pd
 86    >>> df = pd.read_csv('weather.csv')
 87    >>> pipe.sync(df)
 88    ```
 89    """
 90
 91    from ._fetch import (
 92        fetch,
 93        get_backtrack_interval,
 94    )
 95    from ._data import (
 96        get_data,
 97        get_backtrack_data,
 98        get_rowcount,
 99        get_size,
100        get_data,
101        get_doc,
102        get_docs,
103        get_value,
104        _get_data_as_iterator,
105        get_chunk_interval,
106        _scale_delta_to_int_axis,
107        get_chunk_bounds,
108        get_chunk_bounds_batches,
109        parse_date_bounds,
110    )
111    from ._register import register
112    from ._attributes import (
113        attributes,
114        parameters,
115        columns,
116        indices,
117        indexes,
118        dtypes,
119        autoincrement,
120        autotime,
121        upsert,
122        static,
123        tzinfo,
124        enforce,
125        null_indices,
126        mixed_numerics,
127        get_columns,
128        get_columns_types,
129        get_columns_indices,
130        get_indices,
131        get_parameters,
132        get_dtypes,
133        update_parameters,
134        validate_indices,
135        tags,
136        get_id,
137        id,
138        get_val_column,
139        parents,
140        parent,
141        children,
142        child,
143        reference,
144        references,
145        target,
146        _target_legacy,
147        guess_datetime,
148        precision,
149        get_precision,
150    )
151    from ._cache import (
152        _get_cache_connector,
153        _cache_value,
154        _get_cached_value,
155        _invalidate_cache,
156        _get_cache_dir_path,
157        _write_cache_key,
158        _write_cache_file,
159        _write_cache_conn_key,
160        _read_cache_key,
161        _read_cache_file,
162        _read_cache_conn_key,
163        _load_cache_keys,
164        _load_cache_files,
165        _load_cache_conn_keys,
166        _get_cache_keys,
167        _get_cache_file_keys,
168        _get_cache_conn_keys,
169        _clear_cache_key,
170        _clear_cache_file,
171        _clear_cache_conn_key,
172    )
173    from ._show import show
174    from ._edit import edit, edit_definition, update
175    from ._sync import (
176        sync,
177        get_sync_time,
178        exists,
179        filter_existing,
180        _filter_existing_polars,
181        _get_chunk_label,
182        get_num_workers,
183        _persist_new_special_columns,
184    )
185    from ._verify import (
186        verify,
187        get_bound_interval,
188        get_bound_time,
189    )
190    from ._delete import delete
191    from ._drop import drop, drop_indices
192    from ._compress import compress, decompress
193    from ._maintenance import vacuum, analyze, repartition
194    from ._index import create_indices
195    from ._clear import clear
196    from ._deduplicate import deduplicate
197    from ._bootstrap import bootstrap
198    from ._dtypes import enforce_dtypes, infer_dtypes
199    from ._copy import copy_to
200
201    def __init__(
202        self,
203        connector: str = '',
204        metric: str = '',
205        location: Optional[str] = None,
206        parameters: Optional[Dict[str, Any]] = None,
207        columns: Union[Dict[str, str], List[str], None] = None,
208        indices: Optional[Dict[str, Union[str, List[str]]]] = None,
209        tags: Optional[List[str]] = None,
210        target: Optional[str] = None,
211        dtypes: Optional[Dict[str, str]] = None,
212        instance: Optional[Union[str, InstanceConnector]] = None,
213        upsert: Optional[bool] = None,
214        autoincrement: Optional[bool] = None,
215        autotime: Optional[bool] = None,
216        precision: Union[str, Dict[str, Union[str, int]], None] = None,
217        static: Optional[bool] = None,
218        enforce: Optional[bool] = None,
219        null_indices: Optional[bool] = None,
220        mixed_numerics: Optional[bool] = None,
221        compress: Union[bool, Dict[str, Any], None] = None,
222        temporary: bool = False,
223        cache: Optional[bool] = None,
224        cache_connector_keys: Optional[str] = None,
225        references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
226        reference: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
227        parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
228        parent: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
229        children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
230        child: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
231        mrsm_instance: Optional[Union[str, InstanceConnector]] = None,
232        connector_keys: Optional[str] = None,
233        metric_key: Optional[str] = None,
234        location_key: Optional[str] = None,
235        instance_keys: Optional[str] = None,
236        indexes: Union[Dict[str, str], List[str], None] = None,
237        debug: bool = False,
238    ):
239        """
240        Parameters
241        ----------
242        connector: str
243            Keys for the pipe's source connector, e.g. `'sql:main'`.
244
245        metric: str
246            Label for the pipe's contents, e.g. `'weather'`.
247
248        location: str, default None
249            Label for the pipe's location. Defaults to `None`.
250
251        parameters: Optional[Dict[str, Any]], default None
252            Optionally set a pipe's parameters from the constructor,
253            e.g. columns and other attributes.
254            You can edit these parameters with `edit pipes`.
255
256        columns: Union[Dict[str, str], List[str], None], default None
257            Set the `columns` dictionary of `parameters`.
258            If `parameters` is also provided, this dictionary is added under the `'columns'` key.
259
260        indices: Optional[Dict[str, Union[str, List[str]]]], default None
261            Set the `indices` dictionary of `parameters`.
262            If `parameters` is also provided, this dictionary is added under the `'indices'` key.
263
264        tags: Optional[List[str]], default None
265            A list of strings to be added under the `'tags'` key of `parameters`.
266            You can select pipes with certain tags using `--tags`.
267
268        dtypes: Optional[Dict[str, str]], default None
269            Set the `dtypes` dictionary of `parameters`.
270            If `parameters` is also provided, this dictionary is added under the `'dtypes'` key.
271
272        mrsm_instance: Optional[Union[str, InstanceConnector]], default None
273            Connector for the Meerschaum instance where the pipe resides.
274            Defaults to the preconfigured default instance (`'sql:main'`).
275
276        instance: Optional[Union[str, InstanceConnector]], default None
277            Alias for `mrsm_instance`. If `mrsm_instance` is supplied, this value is ignored.
278
279        upsert: Optional[bool], default None
280            If `True`, set `upsert` to `True` in the parameters.
281
282        autoincrement: Optional[bool], default None
283            If `True`, set `autoincrement` in the parameters.
284
285        autotime: Optional[bool], default None
286            If `True`, set `autotime` in the parameters.
287
288        precision: Union[str, Dict[str, Union[str, int]], None], default None
289            If provided, set `precision` in the parameters.
290            This may be either a string (the precision unit) or a dictionary of in the form
291            `{'unit': <unit>, 'interval': <interval>}`.
292            Default is determined by the `datetime` column dtype
293            (e.g. `datetime64[us]` is `microsecond` precision).
294
295        static: Optional[bool], default None
296            If `True`, set `static` in the parameters.
297
298        enforce: Optional[bool], default None
299            If `False`, skip data type enforcement.
300            Default behavior is `True`.
301
302        null_indices: Optional[bool], default None
303            Set to `False` if there will be no null values in the index columns.
304            Defaults to `True`.
305
306        mixed_numerics: bool, default None
307            If `True`, integer columns will be converted to `numeric` when floats are synced.
308            Set to `False` to disable this behavior.
309            Defaults to `True`.
310
311        compress: Union[bool, Dict[str, Any], None], default None
312            If `True` (or a dictionary of compression settings), mark the pipe for compression.
313            For TimescaleDB hypertables, a columnstore (compression) policy is installed
314            automatically on sync. A dictionary may override `segmentby`, `orderby`, and `after`.
315            Defaults to `False`.
316
317        hypercore: bool, default True
318            For TimescaleDB hypertables, enable the Hypercore columnstore at table creation
319            (declaring `segmentby`/`orderby` in `CREATE TABLE`), which causes TimescaleDB to
320            auto-create a columnstore policy. Set to `False` for a plain row-store hypertable.
321            Has no effect unless the pipe is a hypertable (`hypertable`, default `True`).
322
323        temporary: bool, default False
324            If `True`, prevent instance tables (pipes, users, plugins) from being created.
325
326        cache: Optional[bool], default None
327            If `True`, cache the pipe's metadata to disk (in addition to in-memory caching).
328            If `cache` is not explicitly `True`, it is set to `False` if `temporary` is `True`.
329            Defaults to `True` (from `None`).
330
331        cache_connector_keys: Optional[str], default None
332            If provided, use the keys to a Valkey connector (e.g. `valkey:main`).
333
334        references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
335            If provided, inherit the parameters of the reference Pipe(s).
336            May be equal to a string of the Pipe constructor, a dictionary of constructor keys,
337            a Pipe itself, or a list of any of these values.
338
339        parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
340            Set references for parent pipes. See `references` for values.
341
342        children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
343            Set references for child pipes. See `references` for values.
344
345        """
346        from meerschaum.utils.warnings import error, warn
347        if (not connector and not connector_keys) or (not metric and not metric_key):
348            error(
349                "Please provide strings for the connector and metric\n    "
350                + "(first two positional arguments)."
351            )
352
353        ### Fall back to legacy `location_key` just in case.
354        if not location:
355            location = location_key
356
357        if not connector:
358            connector = connector_keys
359
360        if not metric:
361            metric = metric_key
362
363        if location in ('[None]', 'None'):
364            location = None
365
366        from meerschaum._internal.static import STATIC_CONFIG
367        negation_prefix = STATIC_CONFIG['system']['fetch_pipes_keys']['negation_prefix']
368        for k in (connector, metric, location, *(tags or [])):
369            if str(k).startswith(negation_prefix):
370                error(f"A pipe's keys and tags cannot start with the prefix '{negation_prefix}'.")
371
372        self._connector_keys = str(connector)
373        self._connector_key = self.connector_keys ### Alias
374        self._metric_key = metric
375        self._location_key = location
376        self.temporary = temporary
377        self.cache = (
378            cache
379            if cache is not None
380            else ((not temporary) and get_config('pipes', 'cache', 'enabled', warn=False))
381        )
382        self.cache_connector_keys = (
383            str(cache_connector_keys)
384            if cache_connector_keys is not None
385            else None
386        )
387        self.debug = debug
388
389        self._attributes: Dict[str, Any] = {
390            'connector_keys': self._connector_keys,
391            'metric_key': self._metric_key,
392            'location_key': self._location_key,
393            'parameters': {},
394        }
395
396        ### only set parameters if values are provided
397        if isinstance(parameters, dict):
398            self._attributes['parameters'] = parameters
399        else:
400            if parameters is not None:
401                warn(f"The provided parameters are of invalid type '{type(parameters)}'.")
402            self._attributes['parameters'] = {}
403
404        columns = columns or self._attributes.get('parameters', {}).get('columns', None)
405        if isinstance(columns, (list, tuple)):
406            columns = {str(col): str(col) for col in columns}
407        if isinstance(columns, dict):
408            self._attributes['parameters']['columns'] = columns
409        elif isinstance(columns, str) and 'Pipe(' in columns:
410            pass
411        elif columns is not None:
412            warn(f"The provided columns are of invalid type '{type(columns)}'.")
413
414        indices = (
415            indices
416            or indexes
417            or self._attributes.get('parameters', {}).get('indices', None)
418            or self._attributes.get('parameters', {}).get('indexes', None)
419        )
420        if isinstance(indices, dict):
421            indices_key = (
422                'indexes'
423                if 'indexes' in self._attributes['parameters']
424                else 'indices'
425            )
426            self._attributes['parameters'][indices_key] = indices
427
428        if isinstance(tags, (list, tuple)):
429            self._attributes['parameters']['tags'] = tags
430        elif tags is not None:
431            warn(f"The provided tags are of invalid type '{type(tags)}'.")
432
433        if isinstance(target, str):
434            self._attributes['parameters']['target'] = target
435        elif target is not None:
436            warn(f"The provided target is of invalid type '{type(target)}'.")
437
438        if isinstance(dtypes, dict):
439            self._attributes['parameters']['dtypes'] = dtypes
440        elif dtypes is not None:
441            warn(f"The provided dtypes are of invalid type '{type(dtypes)}'.")
442
443        if isinstance(upsert, bool):
444            self._attributes['parameters']['upsert'] = upsert
445
446        if isinstance(autoincrement, bool):
447            self._attributes['parameters']['autoincrement'] = autoincrement
448
449        if isinstance(autotime, bool):
450            self._attributes['parameters']['autotime'] = autotime
451
452        if isinstance(precision, dict):
453            self._attributes['parameters']['precision'] = precision
454        elif isinstance(precision, str):
455            self._attributes['parameters']['precision'] = {'unit': precision}
456
457        if isinstance(static, bool):
458            self._attributes['parameters']['static'] = static
459            self._static = static
460
461        if isinstance(enforce, bool):
462            self._attributes['parameters']['enforce'] = enforce
463
464        if isinstance(null_indices, bool):
465            self._attributes['parameters']['null_indices'] = null_indices
466
467        if isinstance(mixed_numerics, bool):
468            self._attributes['parameters']['mixed_numerics'] = mixed_numerics
469
470        if isinstance(compress, (bool, dict)):
471            self._attributes['parameters']['compress'] = compress
472
473        ### NOTE: The parameters dictionary is {} by default.
474        ###       A Pipe may be registered without parameters, then edited,
475        ###       or a Pipe may be registered with parameters set in-memory first.
476        _mrsm_instance = mrsm_instance if mrsm_instance is not None else (instance or instance_keys)
477        if _mrsm_instance is None:
478            _mrsm_instance = get_config('meerschaum', 'instance', patch=True)
479
480        if not isinstance(_mrsm_instance, str):
481            self._instance_connector = _mrsm_instance
482            self._instance_keys = str(_mrsm_instance)
483        else:
484            self._instance_keys = _mrsm_instance
485
486        if self._instance_keys == 'sql:memory':
487            self.cache = False
488
489        self._cache_locks = collections.defaultdict(lambda: threading.RLock())
490        self._attributes_lock = threading.RLock()
491
492        if references is not None or reference is not None:
493            reference_vals = references if references is not None else reference
494            self.references = reference_vals
495
496        if parents is not None or parent is not None:
497            parent_vals = parents if parents is not None else parent
498            self.parents = parent_vals
499
500        if children is not None or child is not None:
501            children_vals = children if children is not None else child
502            self.children = children_vals
503
504    @property
505    def metric_key(self) -> str:
506        """
507        Return the pipe's metric key.
508        """
509        return self._metric_key
510
511    @property
512    def metric(self) -> str:
513        """
514        Return the pipe's metric key.
515        """
516        return self._metric_key
517
518    @property
519    def location_key(self) -> Union[str, None]:
520        """
521        Return the pipe's location key.
522        """
523        return self._location_key
524
525    @property
526    def location(self) -> Union[str, None]:
527        """
528        Return the pipe's location key.
529        """
530        return self._location_key
531
532    @property
533    def meta(self):
534        """
535        Return the four keys needed to reconstruct this pipe.
536        """
537        return {
538            'connector_keys': self.connector_keys,
539            'metric_key': self.metric_key,
540            'location_key': self.location_key,
541            'instance_keys': self.instance_keys,
542        }
543
544    def keys(self) -> List[str]:
545        """
546        Return the ordered keys for this pipe.
547        """
548        return {
549            key: val
550            for key, val in self.meta.items()
551            if key != 'instance'
552        }
553
554    @property
555    def instance_keys(self) -> str:
556        """
557        Return the pipe's instance keys.
558        """
559        return self._instance_keys
560
561    @property
562    def instance(self) -> Union[InstanceConnector, str]:
563        """
564        Return the pipe's instance connector or keys.
565        """
566        conn = self.instance_connector
567        if conn is None:
568            return self.instance_keys
569        return conn
570
571    @property
572    def instance_connector(self) -> Union[InstanceConnector, None]:
573        """
574        The instance connector on which this pipe resides.
575        """
576        if '_instance_connector' not in self.__dict__:
577            from meerschaum.connectors.parse import parse_instance_keys
578            conn = parse_instance_keys(self.instance_keys)
579            if conn:
580                self._instance_connector = conn
581            else:
582                return None
583        return self._instance_connector
584
585    @property
586    def connector_keys(self) -> str:
587        """
588        Return the pipe's connector keys.
589        """
590        return self._connector_keys
591
592    @property
593    def connector_key(self) -> str:
594        """
595        Legacy: use `Pipe.connector_keys` instead.
596        """
597        return self.connector_keys
598
599    @property
600    def connector(self) -> Union['Connector', str]:
601        """
602        The connector to the data source.
603        """
604        if '_connector' not in self.__dict__:
605            from meerschaum.connectors.parse import parse_instance_keys
606            import warnings
607            with warnings.catch_warnings():
608                warnings.simplefilter('ignore')
609                try:
610                    conn = parse_instance_keys(self.connector_keys)
611                except Exception:
612                    conn = None
613            if conn:
614                self._connector = conn
615            else:
616                return self._connector_keys
617        return self._connector
618
619    def __str__(self, ansi: bool=False):
620        return pipe_repr(self, ansi=ansi)
621
622    def __eq__(self, other):
623        try:
624            return (
625                isinstance(self, type(other))
626                and self.connector_keys == other.connector_keys
627                and self.metric_key == other.metric_key
628                and self.location_key == other.location_key
629                and self.instance_keys == other.instance_keys
630            )
631        except Exception:
632            return False
633
634    def __hash__(self):
635        ### Using an esoteric separator to avoid collisions.
636        sep = "[\"']"
637        return hash(
638            str(self.connector_keys) + sep
639            + str(self.metric_key) + sep
640            + str(self.location_key) + sep
641            + str(self.instance_keys) + sep
642        )
643
644    def __repr__(self, ansi: bool=True, **kw) -> str:
645        if not hasattr(sys, 'ps1'):
646            ansi = False
647
648        return pipe_repr(self, ansi=ansi, **kw)
649
650    def __pt_repr__(self):
651        from meerschaum.utils.packages import attempt_import
652        prompt_toolkit_formatted_text = attempt_import('prompt_toolkit.formatted_text', lazy=False)
653        return prompt_toolkit_formatted_text.ANSI(pipe_repr(self, ansi=True))
654
655    def __getstate__(self) -> Dict[str, Any]:
656        """
657        Define the state dictionary (pickling).
658        """
659        return {
660            'connector_keys': self.connector_keys,
661            'metric_key': self.metric_key,
662            'location_key': self.location_key,
663            'parameters': self._attributes.get('parameters', None),
664            'instance_keys': self.instance_keys,
665        }
666
667    def __setstate__(self, _state: Dict[str, Any]):
668        """
669        Read the state (unpickling).
670        """
671        self.__init__(**_state)
672
673    def __getitem__(self, key: str) -> Any:
674        """
675        Index the pipe's attributes.
676        If the `key` cannot be found`, return `None`.
677        """
678        if key in self.attributes:
679            return self.attributes.get(key, None)
680
681        aliases = {
682            'connector': 'connector_keys',
683            'connector_key': 'connector_keys',
684            'metric': 'metric_key',
685            'location': 'location_key',
686        }
687        aliased_key = aliases.get(key, None)
688        if aliased_key is not None:
689            return self.attributes.get(aliased_key, None)
690
691        property_aliases = {
692            'instance': 'instance_keys',
693            'instance_key': 'instance_keys',
694        }
695        aliased_key = property_aliases.get(key, None)
696        if aliased_key is not None:
697            key = aliased_key
698        return getattr(self, key, None)
699
700    def __copy__(self):
701        """
702        Return a shallow copy of the current pipe.
703        """
704        return mrsm.Pipe(
705            self.connector_keys, self.metric_key, self.location_key,
706            instance=self.instance_keys,
707            parameters=self._attributes.get('parameters', None),
708        )
709
710    def __deepcopy__(self, memo):
711        """
712        Return a deep copy of the current pipe.
713        """
714        return self.__copy__()

Access Meerschaum pipes via Pipe objects.

Pipes are identified by the following:

  1. Connector keys (e.g. 'sql:main')
  2. Metric key (e.g. 'weather')
  3. Location (optional; e.g. None)

A pipe's connector keys correspond to a data source, and when the pipe is synced, its fetch definition is evaluated and executed to produce new data.

Alternatively, new data may be directly synced via pipe.sync():

>>> from meerschaum import Pipe
>>> pipe = Pipe('csv', 'weather')
>>>
>>> import pandas as pd
>>> df = pd.read_csv('weather.csv')
>>> pipe.sync(df)
Pipe( connector: str = '', metric: str = '', location: Optional[str] = None, parameters: Optional[Dict[str, Any]] = None, columns: Union[Dict[str, str], List[str], NoneType] = None, indices: Optional[Dict[str, Union[str, List[str]]]] = None, tags: Optional[List[str]] = None, target: Optional[str] = None, dtypes: Optional[Dict[str, str]] = None, instance: Union[str, InstanceConnector, NoneType] = None, upsert: Optional[bool] = None, autoincrement: Optional[bool] = None, autotime: Optional[bool] = None, precision: Union[str, Dict[str, Union[str, int]], NoneType] = None, static: Optional[bool] = None, enforce: Optional[bool] = None, null_indices: Optional[bool] = None, mixed_numerics: Optional[bool] = None, compress: Union[bool, Dict[str, Any], NoneType] = None, temporary: bool = False, cache: Optional[bool] = None, cache_connector_keys: Optional[str] = None, references: Optional[List[Union[str, Dict[str, Any], Pipe, NoneType]]] = None, reference: Union[str, Dict[str, Any], Pipe, NoneType] = None, parents: Optional[List[Union[str, Dict[str, Any], Pipe, NoneType]]] = None, parent: Union[str, Dict[str, Any], Pipe, NoneType] = None, children: Optional[List[Union[str, Dict[str, Any], Pipe, NoneType]]] = None, child: Union[str, Dict[str, Any], Pipe, NoneType] = None, mrsm_instance: Union[str, InstanceConnector, NoneType] = None, connector_keys: Optional[str] = None, metric_key: Optional[str] = None, location_key: Optional[str] = None, instance_keys: Optional[str] = None, indexes: Union[Dict[str, str], List[str], NoneType] = None, debug: bool = False)
201    def __init__(
202        self,
203        connector: str = '',
204        metric: str = '',
205        location: Optional[str] = None,
206        parameters: Optional[Dict[str, Any]] = None,
207        columns: Union[Dict[str, str], List[str], None] = None,
208        indices: Optional[Dict[str, Union[str, List[str]]]] = None,
209        tags: Optional[List[str]] = None,
210        target: Optional[str] = None,
211        dtypes: Optional[Dict[str, str]] = None,
212        instance: Optional[Union[str, InstanceConnector]] = None,
213        upsert: Optional[bool] = None,
214        autoincrement: Optional[bool] = None,
215        autotime: Optional[bool] = None,
216        precision: Union[str, Dict[str, Union[str, int]], None] = None,
217        static: Optional[bool] = None,
218        enforce: Optional[bool] = None,
219        null_indices: Optional[bool] = None,
220        mixed_numerics: Optional[bool] = None,
221        compress: Union[bool, Dict[str, Any], None] = None,
222        temporary: bool = False,
223        cache: Optional[bool] = None,
224        cache_connector_keys: Optional[str] = None,
225        references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
226        reference: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
227        parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
228        parent: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
229        children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None,
230        child: Union[str, Dict[str, Any], mrsm.Pipe, None] = None,
231        mrsm_instance: Optional[Union[str, InstanceConnector]] = None,
232        connector_keys: Optional[str] = None,
233        metric_key: Optional[str] = None,
234        location_key: Optional[str] = None,
235        instance_keys: Optional[str] = None,
236        indexes: Union[Dict[str, str], List[str], None] = None,
237        debug: bool = False,
238    ):
239        """
240        Parameters
241        ----------
242        connector: str
243            Keys for the pipe's source connector, e.g. `'sql:main'`.
244
245        metric: str
246            Label for the pipe's contents, e.g. `'weather'`.
247
248        location: str, default None
249            Label for the pipe's location. Defaults to `None`.
250
251        parameters: Optional[Dict[str, Any]], default None
252            Optionally set a pipe's parameters from the constructor,
253            e.g. columns and other attributes.
254            You can edit these parameters with `edit pipes`.
255
256        columns: Union[Dict[str, str], List[str], None], default None
257            Set the `columns` dictionary of `parameters`.
258            If `parameters` is also provided, this dictionary is added under the `'columns'` key.
259
260        indices: Optional[Dict[str, Union[str, List[str]]]], default None
261            Set the `indices` dictionary of `parameters`.
262            If `parameters` is also provided, this dictionary is added under the `'indices'` key.
263
264        tags: Optional[List[str]], default None
265            A list of strings to be added under the `'tags'` key of `parameters`.
266            You can select pipes with certain tags using `--tags`.
267
268        dtypes: Optional[Dict[str, str]], default None
269            Set the `dtypes` dictionary of `parameters`.
270            If `parameters` is also provided, this dictionary is added under the `'dtypes'` key.
271
272        mrsm_instance: Optional[Union[str, InstanceConnector]], default None
273            Connector for the Meerschaum instance where the pipe resides.
274            Defaults to the preconfigured default instance (`'sql:main'`).
275
276        instance: Optional[Union[str, InstanceConnector]], default None
277            Alias for `mrsm_instance`. If `mrsm_instance` is supplied, this value is ignored.
278
279        upsert: Optional[bool], default None
280            If `True`, set `upsert` to `True` in the parameters.
281
282        autoincrement: Optional[bool], default None
283            If `True`, set `autoincrement` in the parameters.
284
285        autotime: Optional[bool], default None
286            If `True`, set `autotime` in the parameters.
287
288        precision: Union[str, Dict[str, Union[str, int]], None], default None
289            If provided, set `precision` in the parameters.
290            This may be either a string (the precision unit) or a dictionary of in the form
291            `{'unit': <unit>, 'interval': <interval>}`.
292            Default is determined by the `datetime` column dtype
293            (e.g. `datetime64[us]` is `microsecond` precision).
294
295        static: Optional[bool], default None
296            If `True`, set `static` in the parameters.
297
298        enforce: Optional[bool], default None
299            If `False`, skip data type enforcement.
300            Default behavior is `True`.
301
302        null_indices: Optional[bool], default None
303            Set to `False` if there will be no null values in the index columns.
304            Defaults to `True`.
305
306        mixed_numerics: bool, default None
307            If `True`, integer columns will be converted to `numeric` when floats are synced.
308            Set to `False` to disable this behavior.
309            Defaults to `True`.
310
311        compress: Union[bool, Dict[str, Any], None], default None
312            If `True` (or a dictionary of compression settings), mark the pipe for compression.
313            For TimescaleDB hypertables, a columnstore (compression) policy is installed
314            automatically on sync. A dictionary may override `segmentby`, `orderby`, and `after`.
315            Defaults to `False`.
316
317        hypercore: bool, default True
318            For TimescaleDB hypertables, enable the Hypercore columnstore at table creation
319            (declaring `segmentby`/`orderby` in `CREATE TABLE`), which causes TimescaleDB to
320            auto-create a columnstore policy. Set to `False` for a plain row-store hypertable.
321            Has no effect unless the pipe is a hypertable (`hypertable`, default `True`).
322
323        temporary: bool, default False
324            If `True`, prevent instance tables (pipes, users, plugins) from being created.
325
326        cache: Optional[bool], default None
327            If `True`, cache the pipe's metadata to disk (in addition to in-memory caching).
328            If `cache` is not explicitly `True`, it is set to `False` if `temporary` is `True`.
329            Defaults to `True` (from `None`).
330
331        cache_connector_keys: Optional[str], default None
332            If provided, use the keys to a Valkey connector (e.g. `valkey:main`).
333
334        references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
335            If provided, inherit the parameters of the reference Pipe(s).
336            May be equal to a string of the Pipe constructor, a dictionary of constructor keys,
337            a Pipe itself, or a list of any of these values.
338
339        parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
340            Set references for parent pipes. See `references` for values.
341
342        children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None
343            Set references for child pipes. See `references` for values.
344
345        """
346        from meerschaum.utils.warnings import error, warn
347        if (not connector and not connector_keys) or (not metric and not metric_key):
348            error(
349                "Please provide strings for the connector and metric\n    "
350                + "(first two positional arguments)."
351            )
352
353        ### Fall back to legacy `location_key` just in case.
354        if not location:
355            location = location_key
356
357        if not connector:
358            connector = connector_keys
359
360        if not metric:
361            metric = metric_key
362
363        if location in ('[None]', 'None'):
364            location = None
365
366        from meerschaum._internal.static import STATIC_CONFIG
367        negation_prefix = STATIC_CONFIG['system']['fetch_pipes_keys']['negation_prefix']
368        for k in (connector, metric, location, *(tags or [])):
369            if str(k).startswith(negation_prefix):
370                error(f"A pipe's keys and tags cannot start with the prefix '{negation_prefix}'.")
371
372        self._connector_keys = str(connector)
373        self._connector_key = self.connector_keys ### Alias
374        self._metric_key = metric
375        self._location_key = location
376        self.temporary = temporary
377        self.cache = (
378            cache
379            if cache is not None
380            else ((not temporary) and get_config('pipes', 'cache', 'enabled', warn=False))
381        )
382        self.cache_connector_keys = (
383            str(cache_connector_keys)
384            if cache_connector_keys is not None
385            else None
386        )
387        self.debug = debug
388
389        self._attributes: Dict[str, Any] = {
390            'connector_keys': self._connector_keys,
391            'metric_key': self._metric_key,
392            'location_key': self._location_key,
393            'parameters': {},
394        }
395
396        ### only set parameters if values are provided
397        if isinstance(parameters, dict):
398            self._attributes['parameters'] = parameters
399        else:
400            if parameters is not None:
401                warn(f"The provided parameters are of invalid type '{type(parameters)}'.")
402            self._attributes['parameters'] = {}
403
404        columns = columns or self._attributes.get('parameters', {}).get('columns', None)
405        if isinstance(columns, (list, tuple)):
406            columns = {str(col): str(col) for col in columns}
407        if isinstance(columns, dict):
408            self._attributes['parameters']['columns'] = columns
409        elif isinstance(columns, str) and 'Pipe(' in columns:
410            pass
411        elif columns is not None:
412            warn(f"The provided columns are of invalid type '{type(columns)}'.")
413
414        indices = (
415            indices
416            or indexes
417            or self._attributes.get('parameters', {}).get('indices', None)
418            or self._attributes.get('parameters', {}).get('indexes', None)
419        )
420        if isinstance(indices, dict):
421            indices_key = (
422                'indexes'
423                if 'indexes' in self._attributes['parameters']
424                else 'indices'
425            )
426            self._attributes['parameters'][indices_key] = indices
427
428        if isinstance(tags, (list, tuple)):
429            self._attributes['parameters']['tags'] = tags
430        elif tags is not None:
431            warn(f"The provided tags are of invalid type '{type(tags)}'.")
432
433        if isinstance(target, str):
434            self._attributes['parameters']['target'] = target
435        elif target is not None:
436            warn(f"The provided target is of invalid type '{type(target)}'.")
437
438        if isinstance(dtypes, dict):
439            self._attributes['parameters']['dtypes'] = dtypes
440        elif dtypes is not None:
441            warn(f"The provided dtypes are of invalid type '{type(dtypes)}'.")
442
443        if isinstance(upsert, bool):
444            self._attributes['parameters']['upsert'] = upsert
445
446        if isinstance(autoincrement, bool):
447            self._attributes['parameters']['autoincrement'] = autoincrement
448
449        if isinstance(autotime, bool):
450            self._attributes['parameters']['autotime'] = autotime
451
452        if isinstance(precision, dict):
453            self._attributes['parameters']['precision'] = precision
454        elif isinstance(precision, str):
455            self._attributes['parameters']['precision'] = {'unit': precision}
456
457        if isinstance(static, bool):
458            self._attributes['parameters']['static'] = static
459            self._static = static
460
461        if isinstance(enforce, bool):
462            self._attributes['parameters']['enforce'] = enforce
463
464        if isinstance(null_indices, bool):
465            self._attributes['parameters']['null_indices'] = null_indices
466
467        if isinstance(mixed_numerics, bool):
468            self._attributes['parameters']['mixed_numerics'] = mixed_numerics
469
470        if isinstance(compress, (bool, dict)):
471            self._attributes['parameters']['compress'] = compress
472
473        ### NOTE: The parameters dictionary is {} by default.
474        ###       A Pipe may be registered without parameters, then edited,
475        ###       or a Pipe may be registered with parameters set in-memory first.
476        _mrsm_instance = mrsm_instance if mrsm_instance is not None else (instance or instance_keys)
477        if _mrsm_instance is None:
478            _mrsm_instance = get_config('meerschaum', 'instance', patch=True)
479
480        if not isinstance(_mrsm_instance, str):
481            self._instance_connector = _mrsm_instance
482            self._instance_keys = str(_mrsm_instance)
483        else:
484            self._instance_keys = _mrsm_instance
485
486        if self._instance_keys == 'sql:memory':
487            self.cache = False
488
489        self._cache_locks = collections.defaultdict(lambda: threading.RLock())
490        self._attributes_lock = threading.RLock()
491
492        if references is not None or reference is not None:
493            reference_vals = references if references is not None else reference
494            self.references = reference_vals
495
496        if parents is not None or parent is not None:
497            parent_vals = parents if parents is not None else parent
498            self.parents = parent_vals
499
500        if children is not None or child is not None:
501            children_vals = children if children is not None else child
502            self.children = children_vals
Parameters
  • connector (str): Keys for the pipe's source connector, e.g. 'sql:main'.
  • metric (str): Label for the pipe's contents, e.g. 'weather'.
  • location (str, default None): Label for the pipe's location. Defaults to None.
  • parameters (Optional[Dict[str, Any]], default None): Optionally set a pipe's parameters from the constructor, e.g. columns and other attributes. You can edit these parameters with edit pipes.
  • columns (Union[Dict[str, str], List[str], None], default None): Set the columns dictionary of parameters. If parameters is also provided, this dictionary is added under the 'columns' key.
  • indices (Optional[Dict[str, Union[str, List[str]]]], default None): Set the indices dictionary of parameters. If parameters is also provided, this dictionary is added under the 'indices' key.
  • tags (Optional[List[str]], default None): A list of strings to be added under the 'tags' key of parameters. You can select pipes with certain tags using --tags.
  • dtypes (Optional[Dict[str, str]], default None): Set the dtypes dictionary of parameters. If parameters is also provided, this dictionary is added under the 'dtypes' key.
  • mrsm_instance (Optional[Union[str, InstanceConnector]], default None): Connector for the Meerschaum instance where the pipe resides. Defaults to the preconfigured default instance ('sql:main').
  • instance (Optional[Union[str, InstanceConnector]], default None): Alias for mrsm_instance. If mrsm_instance is supplied, this value is ignored.
  • upsert (Optional[bool], default None): If True, set upsert to True in the parameters.
  • autoincrement (Optional[bool], default None): If True, set autoincrement in the parameters.
  • autotime (Optional[bool], default None): If True, set autotime in the parameters.
  • precision (Union[str, Dict[str, Union[str, int]], None], default None): If provided, set precision in the parameters. This may be either a string (the precision unit) or a dictionary of in the form {'unit': <unit>, 'interval': <interval>}. Default is determined by the datetime column dtype (e.g. datetime64[us] is microsecond precision).
  • static (Optional[bool], default None): If True, set static in the parameters.
  • enforce (Optional[bool], default None): If False, skip data type enforcement. Default behavior is True.
  • null_indices (Optional[bool], default None): Set to False if there will be no null values in the index columns. Defaults to True.
  • mixed_numerics (bool, default None): If True, integer columns will be converted to numeric when floats are synced. Set to False to disable this behavior. Defaults to True.
  • compress (Union[bool, Dict[str, Any], None], default None): If True (or a dictionary of compression settings), mark the pipe for compression. For TimescaleDB hypertables, a columnstore (compression) policy is installed automatically on sync. A dictionary may override segmentby, orderby, and after. Defaults to False.
  • hypercore (bool, default True): For TimescaleDB hypertables, enable the Hypercore columnstore at table creation (declaring segmentby/orderby in CREATE TABLE), which causes TimescaleDB to auto-create a columnstore policy. Set to False for a plain row-store hypertable. Has no effect unless the pipe is a hypertable (hypertable, default True).
  • temporary (bool, default False): If True, prevent instance tables (pipes, users, plugins) from being created.
  • cache (Optional[bool], default None): If True, cache the pipe's metadata to disk (in addition to in-memory caching). If cache is not explicitly True, it is set to False if temporary is True. Defaults to True (from None).
  • cache_connector_keys (Optional[str], default None): If provided, use the keys to a Valkey connector (e.g. valkey:main).
  • references (Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None): If provided, inherit the parameters of the reference Pipe(s). May be equal to a string of the Pipe constructor, a dictionary of constructor keys, a Pipe itself, or a list of any of these values.
  • parents (Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None): Set references for parent pipes. See references for values.
  • children (Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None): Set references for child pipes. See references for values.
temporary
cache
cache_connector_keys
debug
metric_key: str
504    @property
505    def metric_key(self) -> str:
506        """
507        Return the pipe's metric key.
508        """
509        return self._metric_key

Return the pipe's metric key.

metric: str
511    @property
512    def metric(self) -> str:
513        """
514        Return the pipe's metric key.
515        """
516        return self._metric_key

Return the pipe's metric key.

location_key: Optional[str]
518    @property
519    def location_key(self) -> Union[str, None]:
520        """
521        Return the pipe's location key.
522        """
523        return self._location_key

Return the pipe's location key.

location: Optional[str]
525    @property
526    def location(self) -> Union[str, None]:
527        """
528        Return the pipe's location key.
529        """
530        return self._location_key

Return the pipe's location key.

meta
532    @property
533    def meta(self):
534        """
535        Return the four keys needed to reconstruct this pipe.
536        """
537        return {
538            'connector_keys': self.connector_keys,
539            'metric_key': self.metric_key,
540            'location_key': self.location_key,
541            'instance_keys': self.instance_keys,
542        }

Return the four keys needed to reconstruct this pipe.

def keys(self) -> List[str]:
544    def keys(self) -> List[str]:
545        """
546        Return the ordered keys for this pipe.
547        """
548        return {
549            key: val
550            for key, val in self.meta.items()
551            if key != 'instance'
552        }

Return the ordered keys for this pipe.

instance_keys: str
554    @property
555    def instance_keys(self) -> str:
556        """
557        Return the pipe's instance keys.
558        """
559        return self._instance_keys

Return the pipe's instance keys.

instance: Union[InstanceConnector, str]
561    @property
562    def instance(self) -> Union[InstanceConnector, str]:
563        """
564        Return the pipe's instance connector or keys.
565        """
566        conn = self.instance_connector
567        if conn is None:
568            return self.instance_keys
569        return conn

Return the pipe's instance connector or keys.

instance_connector: Optional[InstanceConnector]
571    @property
572    def instance_connector(self) -> Union[InstanceConnector, None]:
573        """
574        The instance connector on which this pipe resides.
575        """
576        if '_instance_connector' not in self.__dict__:
577            from meerschaum.connectors.parse import parse_instance_keys
578            conn = parse_instance_keys(self.instance_keys)
579            if conn:
580                self._instance_connector = conn
581            else:
582                return None
583        return self._instance_connector

The instance connector on which this pipe resides.

connector_keys: str
585    @property
586    def connector_keys(self) -> str:
587        """
588        Return the pipe's connector keys.
589        """
590        return self._connector_keys

Return the pipe's connector keys.

connector_key: str
592    @property
593    def connector_key(self) -> str:
594        """
595        Legacy: use `Pipe.connector_keys` instead.
596        """
597        return self.connector_keys

Legacy: use Pipe.connector_keys instead.

connector: "Union['Connector', str]"
599    @property
600    def connector(self) -> Union['Connector', str]:
601        """
602        The connector to the data source.
603        """
604        if '_connector' not in self.__dict__:
605            from meerschaum.connectors.parse import parse_instance_keys
606            import warnings
607            with warnings.catch_warnings():
608                warnings.simplefilter('ignore')
609                try:
610                    conn = parse_instance_keys(self.connector_keys)
611                except Exception:
612                    conn = None
613            if conn:
614                self._connector = conn
615            else:
616                return self._connector_keys
617        return self._connector

The connector to the data source.

def fetch( self, begin: Union[datetime.datetime, int, str, NoneType] = '', end: Union[datetime.datetime, int, NoneType] = None, check_existing: bool = True, sync_chunks: bool = False, debug: bool = False, **kw: Any) -> Union[pandas.DataFrame, Iterator[pandas.DataFrame], NoneType]:
21def fetch(
22    self,
23    begin: Union[datetime, int, str, None] = '',
24    end: Union[datetime, int, None] = None,
25    check_existing: bool = True,
26    sync_chunks: bool = False,
27    debug: bool = False,
28    **kw: Any
29) -> Union['pd.DataFrame', Iterator['pd.DataFrame'], None]:
30    """
31    Fetch a Pipe's latest data from its connector.
32
33    Parameters
34    ----------
35    begin: Union[datetime, str, None], default '':
36        If provided, only fetch data newer than or equal to `begin`.
37
38    end: Optional[datetime], default None:
39        If provided, only fetch data older than or equal to `end`.
40
41    check_existing: bool, default True
42        If `False`, do not apply the backtrack interval.
43
44    sync_chunks: bool, default False
45        If `True` and the pipe's connector is of type `'sql'`, begin syncing chunks while fetching
46        loads chunks into memory.
47
48    debug: bool, default False
49        Verbosity toggle.
50
51    Returns
52    -------
53    A `pd.DataFrame` of the newest unseen data.
54
55    """
56    if 'fetch' not in dir(self.connector):
57        warn(f"No `fetch()` function defined for connector '{self.connector}'")
58        return None
59
60    from meerschaum.connectors import get_connector_plugin
61    from meerschaum.utils.misc import filter_arguments
62
63    _chunk_hook = kw.pop('chunk_hook', None)
64    kw['workers'] = self.get_num_workers(kw.get('workers', None))
65    if sync_chunks and _chunk_hook is None:
66
67        def _chunk_hook(chunk, **_kw) -> SuccessTuple:
68            """
69            Wrap `Pipe.sync()` with a custom chunk label prepended to the message.
70            """
71            from meerschaum.config._patch import apply_patch_to_config
72            kwargs = apply_patch_to_config(kw, _kw)
73            chunk_success, chunk_message = self.sync(chunk, **kwargs)
74            chunk_label = self._get_chunk_label(chunk, self.columns.get('datetime', None))
75            if chunk_label:
76                chunk_message = '\n' + chunk_label + '\n' + chunk_message
77            return chunk_success, chunk_message
78
79    begin, end = self.parse_date_bounds(begin, end)
80
81    with mrsm.Venv(get_connector_plugin(self.connector)):
82        _args, _kwargs = filter_arguments(
83            self.connector.fetch,
84            self,
85            begin=_determine_begin(
86                self,
87                begin,
88                end,
89                check_existing=check_existing,
90                debug=debug,
91            ),
92            end=end,
93            chunk_hook=_chunk_hook,
94            debug=debug,
95            **kw
96        )
97        df = self.connector.fetch(*_args, **_kwargs)
98    return df

Fetch a Pipe's latest data from its connector.

Parameters
  • begin (Union[datetime, str, None], default '':): If provided, only fetch data newer than or equal to begin.
  • end (Optional[datetime], default None:): If provided, only fetch data older than or equal to end.
  • check_existing (bool, default True): If False, do not apply the backtrack interval.
  • sync_chunks (bool, default False): If True and the pipe's connector is of type 'sql', begin syncing chunks while fetching loads chunks into memory.
  • debug (bool, default False): Verbosity toggle.
Returns
  • A pd.DataFrame of the newest unseen data.
def get_backtrack_interval( self, check_existing: bool = True, debug: bool = False) -> Union[datetime.timedelta, int]:
101def get_backtrack_interval(
102    self,
103    check_existing: bool = True,
104    debug: bool = False,
105) -> Union[timedelta, int]:
106    """
107    Get the chunk interval to use for this pipe.
108
109    Parameters
110    ----------
111    check_existing: bool, default True
112        If `False`, return a backtrack_interval of 0 minutes.
113
114    Returns
115    -------
116    The backtrack interval (`timedelta` or `int`) to use with this pipe's `datetime` axis.
117    """
118    from meerschaum.utils.dtypes import MRSM_PRECISION_UNITS_SCALARS, MRSM_PRECISION_UNITS_ALIASES
119    default_backtrack_minutes = get_config('pipes', 'parameters', 'fetch', 'backtrack_minutes')
120    configured_backtrack_minutes = self.parameters.get('fetch', {}).get('backtrack_minutes', None)
121    backtrack_minutes = (
122        configured_backtrack_minutes
123        if configured_backtrack_minutes is not None
124        else default_backtrack_minutes
125    ) if check_existing else 0
126
127    dt_col = self.columns.get('datetime', None)
128    if dt_col is None:
129        return timedelta(minutes=backtrack_minutes)
130
131    dt_dtype = self.dtypes.get(dt_col, 'datetime')
132    if 'int' in dt_dtype.lower():
133        if not self.parameters.get('precision', None):
134            return backtrack_minutes
135        precision_unit = self.precision.get('unit', None)
136        true_unit = MRSM_PRECISION_UNITS_ALIASES.get(precision_unit, precision_unit)
137        scalar = MRSM_PRECISION_UNITS_SCALARS.get(true_unit, None)
138        if scalar is not None:
139            return int(backtrack_minutes * 60 * scalar)
140        return backtrack_minutes
141
142    return timedelta(minutes=backtrack_minutes)

Get the chunk interval to use for this pipe.

Parameters
  • check_existing (bool, default True): If False, return a backtrack_interval of 0 minutes.
Returns
  • The backtrack interval (timedelta or int) to use with this pipe's datetime axis.
def get_data( self, select_columns: Optional[List[str]] = None, omit_columns: Optional[List[str]] = None, begin: Union[datetime.datetime, int, str, NoneType] = None, end: Union[datetime.datetime, int, str, NoneType] = None, params: Optional[Dict[str, Any]] = None, as_docs: bool = False, as_iterator: bool = False, as_chunks: bool = False, as_dask: bool = False, as_polars: bool = False, add_missing_columns: bool = False, chunk_interval: Union[datetime.timedelta, int, NoneType] = None, order: Optional[str] = 'asc', limit: Optional[int] = None, fresh: bool = False, debug: bool = False, **kw: Any) -> Union[pandas.DataFrame, Iterator[pandas.DataFrame], NoneType]:
 23def get_data(
 24    self,
 25    select_columns: Optional[List[str]] = None,
 26    omit_columns: Optional[List[str]] = None,
 27    begin: Union[datetime, int, str, None] = None,
 28    end: Union[datetime, int, str, None] = None,
 29    params: Optional[Dict[str, Any]] = None,
 30    as_docs: bool = False,
 31    as_iterator: bool = False,
 32    as_chunks: bool = False,
 33    as_dask: bool = False,
 34    as_polars: bool = False,
 35    add_missing_columns: bool = False,
 36    chunk_interval: Union[timedelta, int, None] = None,
 37    order: Optional[str] = 'asc',
 38    limit: Optional[int] = None,
 39    fresh: bool = False,
 40    debug: bool = False,
 41    **kw: Any
 42) -> Union['pd.DataFrame', Iterator['pd.DataFrame'], None]:
 43    """
 44    Get a pipe's data from the instance connector.
 45
 46    Parameters
 47    ----------
 48    select_columns: Optional[List[str]], default None
 49        If provided, only select these given columns.
 50        Otherwise select all available columns (i.e. `SELECT *`).
 51
 52    omit_columns: Optional[List[str]], default None
 53        If provided, remove these columns from the selection.
 54
 55    begin: Union[datetime, int, str, None], default None
 56        Lower bound datetime to begin searching for data (inclusive).
 57        Translates to a `WHERE` clause like `WHERE datetime >= begin`.
 58        Defaults to `None`.
 59
 60    end: Union[datetime, int, str, None], default None
 61        Upper bound datetime to stop searching for data (inclusive).
 62        Translates to a `WHERE` clause like `WHERE datetime < end`.
 63        Defaults to `None`.
 64
 65    params: Optional[Dict[str, Any]], default None
 66        Filter the retrieved data by a dictionary of parameters.
 67        See `meerschaum.utils.sql.build_where` for more details. 
 68
 69    as_docs: bool, default False
 70        If `True`, return a list of dictionaries rather than a DataFrame.
 71        Relies on `get_pipe_docs` from the instance connector if implemented.
 72        May be combined with `as_chunks` to return an `Iterator[List[Dict]]`
 73        chunked by time bounds (useful for large result sets without pandas overhead).
 74
 75    as_iterator: bool, default False
 76        If `True`, return a generator of chunks of pipe data.
 77        When combined with `as_docs=True`, yields `List[Dict]` per chunk instead of DataFrames.
 78
 79    as_chunks: bool, default False
 80        Alias for `as_iterator`.
 81        When combined with `as_docs=True`, yields `List[Dict]` per chunk instead of DataFrames.
 82
 83    as_dask: bool, default False
 84        If `True`, return a `dask.DataFrame`
 85        (which may be loaded into a Pandas DataFrame with `df.compute()`).
 86
 87    as_polars: bool, default False
 88        If `True`, return a `polars.DataFrame`. When combined with `as_iterator=True`,
 89        yield Polars DataFrames. This may not be combined with `as_docs` or `as_dask`.
 90
 91    add_missing_columns: bool, default False
 92        If `True`, add any missing columns from `Pipe.dtypes` to the dataframe.
 93
 94    chunk_interval: Union[timedelta, int, None], default None
 95        If `as_iterator`, then return chunks with `begin` and `end` separated by this interval.
 96        This may be set under `pipe.parameters['chunk_minutes']`.
 97        By default, use a timedelta of 43200 minutes (30 days).
 98        If `chunk_interval` is an integer and the `datetime` axis a timestamp,
 99        then use a timedelta with the number of minutes configured to this value.
100        If the `datetime` axis is an integer, an `int` is used verbatim as the number of
101        epoch units, and a `timedelta` is scaled into epoch units by the pipe's `precision`.
102
103    order: Optional[str], default 'asc'
104        If `order` is not `None`, sort the resulting dataframe by indices.
105
106    limit: Optional[int], default None
107        If provided, cap the dataframe to this many rows.
108
109    fresh: bool, default False
110        If `True`, skip local cache and directly query the instance connector.
111
112    debug: bool, default False
113        Verbosity toggle.
114        Defaults to `False`.
115
116    Returns
117    -------
118    A `pd.DataFrame` of the pipe's data (default).
119    A `List[Dict]` if `as_docs=True`.
120    An `Iterator[pd.DataFrame]` if `as_chunks=True` (or `as_iterator=True`).
121    An `Iterator[List[Dict]]` if both `as_docs=True` and `as_chunks=True`.
122    A `polars.DataFrame` if `as_polars=True`.
123
124    """
125    from meerschaum.utils.warnings import warn
126    from meerschaum.utils.venv import Venv
127    from meerschaum.connectors import get_connector_plugin
128    from meerschaum.utils.dtypes import to_pandas_dtype
129    from meerschaum.utils.dataframe import add_missing_cols_to_df, df_is_chunk_generator
130    from meerschaum.utils.packages import attempt_import
131    from meerschaum.utils.warnings import dprint
132    dd = attempt_import('dask.dataframe') if as_dask else None
133    dask = attempt_import('dask') if as_dask else None
134    _ = attempt_import('partd', lazy=False) if as_dask else None
135
136    if as_polars and (as_docs or as_dask):
137        raise ValueError("`as_polars` may not be combined with `as_docs` or `as_dask`.")
138
139    if select_columns == '*':
140        select_columns = None
141    elif isinstance(select_columns, str):
142        select_columns = [select_columns]
143
144    if isinstance(omit_columns, str):
145        omit_columns = [omit_columns]
146
147    begin, end = self.parse_date_bounds(begin, end, debug=debug)
148    as_iterator = as_iterator or as_chunks
149    dt_col = self.columns.get('datetime', None)
150
151    def _sort_df(_df):
152        if df_is_chunk_generator(_df):
153            return _df
154        indices = [] if dt_col not in _df.columns else [dt_col]
155        non_dt_cols = [
156            col
157            for col_ix, col in self.columns.items()
158            if col_ix != 'datetime' and col in _df.columns
159        ]
160        indices.extend(non_dt_cols)
161        if _df.__module__.split('.')[0] == 'polars':
162            _df = _df.sort(
163                indices,
164                descending=(str(order).lower() != 'asc'),
165            )
166        elif 'dask' not in _df.__module__:
167            ascending = str(order).lower() == 'asc'
168            dt_series = _df[dt_col] if indices and indices[0] == dt_col else None
169            ### Unique, non-null, ordered datetimes (e.g. from the SQL `ORDER BY`) are already sorted.
170            already_sorted = (
171                dt_series is not None
172                and not dt_series.hasnans
173                and (
174                    dt_series.is_monotonic_increasing
175                    if ascending
176                    else dt_series.is_monotonic_decreasing
177                )
178                and dt_series.is_unique
179            )
180            if not already_sorted:
181                _df.sort_values(
182                    by=indices,
183                    inplace=True,
184                    ascending=ascending,
185                )
186            _df.reset_index(drop=True, inplace=True)
187        else:
188            _df = _df.sort_values(
189                by=indices,
190                ascending=(str(order).lower() == 'asc'),
191            )
192            _df = _df.reset_index(drop=True)
193        if limit is not None and len(_df) > limit:
194            return _df.head(limit)
195        return _df
196
197    if as_iterator or as_chunks:
198        df = self._get_data_as_iterator(
199            select_columns=select_columns,
200            omit_columns=omit_columns,
201            begin=begin,
202            end=end,
203            params=params,
204            chunk_interval=chunk_interval,
205            limit=limit,
206            order=order,
207            as_docs=as_docs,
208            as_polars=as_polars,
209            fresh=fresh,
210            debug=debug,
211        )
212        if as_docs:
213            return df
214        df = _sort_df(df)
215        return df
216
217    if as_dask:
218        from multiprocessing.pool import ThreadPool
219        dask_pool = ThreadPool(self.get_num_workers())
220        dask.config.set(pool=dask_pool)
221        chunk_interval = self.get_chunk_interval(chunk_interval, debug=debug)
222        bounds = self.get_chunk_bounds(
223            begin=begin,
224            end=end,
225            bounded=False,
226            chunk_interval=chunk_interval,
227            debug=debug,
228        )
229        dask_chunks = [
230            dask.delayed(self.get_data)(
231                select_columns=select_columns,
232                omit_columns=omit_columns,
233                begin=chunk_begin,
234                end=chunk_end,
235                params=params,
236                chunk_interval=chunk_interval,
237                order=order,
238                limit=limit,
239                fresh=fresh,
240                add_missing_columns=True,
241                debug=debug,
242            )
243            for (chunk_begin, chunk_end) in bounds
244        ]
245        dask_meta = {
246            col: to_pandas_dtype(typ)
247            for col, typ in self.get_dtypes(refresh=True, infer=True, debug=debug).items()
248        }
249        if debug:
250            dprint(f"Dask meta:\n{dask_meta}")
251        dask_df = dd.from_delayed(dask_chunks, meta=dask_meta, verify_meta=False)
252        return _sort_df(dask_df.astype(dask_meta))
253
254    if not self.exists(debug=debug):
255        return [] if as_docs else None
256
257    if as_docs:
258        with Venv(get_connector_plugin(self.instance_connector)):
259            docs = self.instance_connector.get_pipe_docs(
260                pipe=self,
261                select_columns=select_columns,
262                omit_columns=omit_columns,
263                begin=begin,
264                end=end,
265                params=params,
266                limit=limit,
267                order=order,
268                debug=debug,
269                **kw
270            )
271        return docs if docs is not None else []
272
273    with Venv(get_connector_plugin(self.instance_connector)):
274        df = self.instance_connector.get_pipe_data(
275            pipe=self,
276            select_columns=select_columns,
277            omit_columns=omit_columns,
278            begin=begin,
279            end=end,
280            params=params,
281            limit=limit,
282            order=order,
283            debug=debug,
284            **(
285                {'as_polars': True}
286                if as_polars and self.instance_connector.type == 'sql'
287                else {}
288            ),
289            **kw
290        )
291        if df is None:
292            return df
293
294        if not select_columns:
295            select_columns = [col for col in df.columns]
296
297        pipe_dtypes = self.get_dtypes(refresh=False, debug=debug)
298        cols_to_omit = [
299            col
300            for col in df.columns
301            if (
302                col in (omit_columns or [])
303                or
304                col not in (select_columns or [])
305            )
306        ]
307        cols_to_add = [
308            col
309            for col in select_columns
310            if col not in df.columns
311        ] + ([
312            col
313            for col in pipe_dtypes
314            if col not in df.columns
315        ] if add_missing_columns else [])
316        if cols_to_omit:
317            warn(
318                (
319                    f"Received {len(cols_to_omit)} omitted column"
320                    + ('s' if len(cols_to_omit) != 1 else '')
321                    + f" for {self}. "
322                    + "Consider adding `select_columns` and `omit_columns` support to "
323                    + f"'{self.instance_connector.type}' connectors to improve performance."
324                ),
325                stack=False,
326            )
327            _cols_to_select = [col for col in df.columns if col not in cols_to_omit]
328            df = df[_cols_to_select]
329
330        if cols_to_add:
331            if not add_missing_columns:
332                from meerschaum.utils.misc import items_str
333                warn(
334                    f"Will add columns {items_str(cols_to_add)} as nulls to dataframe.",
335                    stack=False,
336                )
337
338            df = add_missing_cols_to_df(
339                df,
340                {
341                    col: pipe_dtypes.get(col, 'string')
342                    for col in cols_to_add
343                },
344            )
345
346        enforced_df = self.enforce_dtypes(
347            df,
348            dtypes=pipe_dtypes,
349            as_polars=as_polars,
350            debug=debug,
351        )
352
353        if order:
354            enforced_df = _sort_df(enforced_df)
355        return enforced_df

Get a pipe's data from the instance connector.

Parameters
  • select_columns (Optional[List[str]], default None): If provided, only select these given columns. Otherwise select all available columns (i.e. SELECT *).
  • omit_columns (Optional[List[str]], default None): If provided, remove these columns from the selection.
  • begin (Union[datetime, int, str, None], default None): Lower bound datetime to begin searching for data (inclusive). Translates to a WHERE clause like WHERE datetime >= begin. Defaults to None.
  • end (Union[datetime, int, str, None], default None): Upper bound datetime to stop searching for data (inclusive). Translates to a WHERE clause like WHERE datetime < end. Defaults to None.
  • params (Optional[Dict[str, Any]], default None): Filter the retrieved data by a dictionary of parameters. See meerschaum.utils.sql.build_where for more details.
  • as_docs (bool, default False): If True, return a list of dictionaries rather than a DataFrame. Relies on get_pipe_docs from the instance connector if implemented. May be combined with as_chunks to return an Iterator[List[Dict]] chunked by time bounds (useful for large result sets without pandas overhead).
  • as_iterator (bool, default False): If True, return a generator of chunks of pipe data. When combined with as_docs=True, yields List[Dict] per chunk instead of DataFrames.
  • as_chunks (bool, default False): Alias for as_iterator. When combined with as_docs=True, yields List[Dict] per chunk instead of DataFrames.
  • as_dask (bool, default False): If True, return a dask.DataFrame (which may be loaded into a Pandas DataFrame with df.compute()).
  • as_polars (bool, default False): If True, return a polars.DataFrame. When combined with as_iterator=True, yield Polars DataFrames. This may not be combined with as_docs or as_dask.
  • add_missing_columns (bool, default False): If True, add any missing columns from Pipe.dtypes to the dataframe.
  • chunk_interval (Union[timedelta, int, None], default None): If as_iterator, then return chunks with begin and end separated by this interval. This may be set under pipe.parameters['chunk_minutes']. By default, use a timedelta of 43200 minutes (30 days). If chunk_interval is an integer and the datetime axis a timestamp, then use a timedelta with the number of minutes configured to this value. If the datetime axis is an integer, an int is used verbatim as the number of epoch units, and a timedelta is scaled into epoch units by the pipe's precision.
  • order (Optional[str], default 'asc'): If order is not None, sort the resulting dataframe by indices.
  • limit (Optional[int], default None): If provided, cap the dataframe to this many rows.
  • fresh (bool, default False): If True, skip local cache and directly query the instance connector.
  • debug (bool, default False): Verbosity toggle. Defaults to False.
Returns
  • A pd.DataFrame of the pipe's data (default).
  • A List[Dict] if as_docs=True.
  • An Iterator[pd.DataFrame] if as_chunks=True (or as_iterator=True).
  • An Iterator[List[Dict]] if both as_docs=True and as_chunks=True.
  • A polars.DataFrame if as_polars=True.
def get_backtrack_data( self, backtrack_minutes: Optional[int] = None, begin: Union[datetime.datetime, int, NoneType] = None, params: Optional[Dict[str, Any]] = None, limit: Optional[int] = None, fresh: bool = False, debug: bool = False, **kw: Any) -> Optional[pandas.DataFrame]:
453def get_backtrack_data(
454    self,
455    backtrack_minutes: Optional[int] = None,
456    begin: Union[datetime, int, None] = None,
457    params: Optional[Dict[str, Any]] = None,
458    limit: Optional[int] = None,
459    fresh: bool = False,
460    debug: bool = False,
461    **kw: Any
462) -> Optional['pd.DataFrame']:
463    """
464    Get the most recent data from the instance connector as a Pandas DataFrame.
465
466    Parameters
467    ----------
468    backtrack_minutes: Optional[int], default None
469        How many minutes from `begin` to select from.
470        If `None`, use `pipe.parameters['fetch']['backtrack_minutes']`.
471
472    begin: Optional[datetime], default None
473        The starting point to search for data.
474        If begin is `None` (default), use the most recent observed datetime
475        (AKA sync_time).
476
477        ```
478        E.g. begin = 02:00
479
480        Search this region.           Ignore this, even if there's data.
481        /  /  /  /  /  /  /  /  /  |
482        -----|----------|----------|----------|----------|----------|
483        00:00      01:00      02:00      03:00      04:00      05:00
484
485        ```
486
487    params: Optional[Dict[str, Any]], default None
488        The standard Meerschaum `params` query dictionary.
489
490    limit: Optional[int], default None
491        If provided, cap the number of rows to be returned.
492
493    fresh: bool, default False
494        If `True`, Ignore local cache and pull directly from the instance connector.
495        Only comes into effect if a pipe was created with `cache=True`.
496
497    debug: bool default False
498        Verbosity toggle.
499
500    Returns
501    -------
502    A `pd.DataFrame` for the pipe's data corresponding to the provided parameters. Backtrack data
503    is a convenient way to get a pipe's data "backtracked" from the most recent datetime.
504    """
505    from meerschaum.utils.venv import Venv
506    from meerschaum.connectors import get_connector_plugin
507
508    if not self.exists(debug=debug):
509        return None
510
511    begin = self.parse_date_bounds(begin, debug=debug)
512
513    backtrack_interval = self.get_backtrack_interval(debug=debug)
514    if backtrack_minutes is None:
515        backtrack_minutes = (
516            (backtrack_interval.total_seconds() / 60)
517            if isinstance(backtrack_interval, timedelta)
518            else backtrack_interval
519        )
520
521    if hasattr(self.instance_connector, 'get_backtrack_data'):
522        with Venv(get_connector_plugin(self.instance_connector)):
523            return self.enforce_dtypes(
524                self.instance_connector.get_backtrack_data(
525                    pipe=self,
526                    begin=begin,
527                    backtrack_minutes=backtrack_minutes,
528                    params=params,
529                    limit=limit,
530                    debug=debug,
531                    **kw
532                ),
533                debug=debug,
534            )
535
536    if begin is None:
537        begin = self.get_sync_time(params=params, debug=debug)
538
539    backtrack_interval = (
540        timedelta(minutes=backtrack_minutes)
541        if isinstance(begin, datetime)
542        else backtrack_minutes
543    )
544    if begin is not None:
545        begin = begin - backtrack_interval
546
547    kw['order'] = kw.get('order', 'desc') or 'desc'
548    return self.get_data(
549        begin=begin,
550        params=params,
551        debug=debug,
552        limit=limit,
553        **kw
554    )

Get the most recent data from the instance connector as a Pandas DataFrame.

Parameters
  • backtrack_minutes (Optional[int], default None): How many minutes from begin to select from. If None, use pipe.parameters['fetch']['backtrack_minutes'].
  • begin (Optional[datetime], default None): The starting point to search for data. If begin is None (default), use the most recent observed datetime (AKA sync_time).

    E.g. begin = 02:00
    
    Search this region.           Ignore this, even if there's data.
    /  /  /  /  /  /  /  /  /  |
    -----|----------|----------|----------|----------|----------|
    00:00      01:00      02:00      03:00      04:00      05:00
    
    
  • params (Optional[Dict[str, Any]], default None): The standard Meerschaum params query dictionary.

  • limit (Optional[int], default None): If provided, cap the number of rows to be returned.
  • fresh (bool, default False): If True, Ignore local cache and pull directly from the instance connector. Only comes into effect if a pipe was created with cache=True.
  • debug (bool default False): Verbosity toggle.
Returns
  • A pd.DataFrame for the pipe's data corresponding to the provided parameters. Backtrack data
  • is a convenient way to get a pipe's data "backtracked" from the most recent datetime.
def get_rowcount( self, begin: Union[datetime.datetime, int, NoneType] = None, end: Union[datetime.datetime, int, NoneType] = None, params: Optional[Dict[str, Any]] = None, remote: bool = False, debug: bool = False) -> int:
557def get_rowcount(
558    self,
559    begin: Union[datetime, int, None] = None,
560    end: Union[datetime, int, None] = None,
561    params: Optional[Dict[str, Any]] = None,
562    remote: bool = False,
563    debug: bool = False
564) -> int:
565    """
566    Get a Pipe's instance or remote rowcount.
567
568    Parameters
569    ----------
570    begin: Optional[datetime], default None
571        Count rows where datetime > begin.
572
573    end: Optional[datetime], default None
574        Count rows where datetime < end.
575
576    remote: bool, default False
577        Count rows from a pipe's remote source.
578        **NOTE**: This is experimental!
579
580    debug: bool, default False
581        Verbosity toggle.
582
583    Returns
584    -------
585    An `int` of the number of rows in the pipe corresponding to the provided parameters.
586    Returned 0 if the pipe does not exist.
587    """
588    from meerschaum.utils.warnings import warn
589    from meerschaum.utils.venv import Venv
590    from meerschaum.connectors import get_connector_plugin
591    from meerschaum.utils.misc import filter_keywords
592
593    begin, end = self.parse_date_bounds(begin, end, debug=debug)
594    connector = self.instance_connector if not remote else self.connector
595    try:
596        with Venv(get_connector_plugin(connector)):
597            if not hasattr(connector, 'get_pipe_rowcount'):
598                warn(
599                    f"Connectors of type '{connector.type}' "
600                    "do not implement `get_pipe_rowcount()`.",
601                    stack=False,
602                )
603                return 0
604            kwargs = filter_keywords(
605                connector.get_pipe_rowcount,
606                begin=begin,
607                end=end,
608                params=params,
609                remote=remote,
610                debug=debug,
611            )
612            if remote and 'remote' not in kwargs:
613                warn(
614                    f"Connectors of type '{connector.type}' do not support remote rowcounts.",
615                    stack=False,
616                )
617                return 0
618            rowcount = connector.get_pipe_rowcount(
619                self,
620                begin=begin,
621                end=end,
622                params=params,
623                remote=remote,
624                debug=debug,
625            )
626            if rowcount is None:
627                return 0
628            return rowcount
629    except AttributeError as e:
630        warn(e)
631        if remote:
632            return 0
633    warn(f"Failed to get a rowcount for {self}.")
634    return 0

Get a Pipe's instance or remote rowcount.

Parameters
  • begin (Optional[datetime], default None): Count rows where datetime > begin.
  • end (Optional[datetime], default None): Count rows where datetime < end.
  • remote (bool, default False): Count rows from a pipe's remote source. NOTE: This is experimental!
  • debug (bool, default False): Verbosity toggle.
Returns
  • An int of the number of rows in the pipe corresponding to the provided parameters.
  • Returned 0 if the pipe does not exist.
def get_size(self, debug: bool = False, **kw: Any) -> Optional[int]:
637def get_size(
638    self,
639    debug: bool = False,
640    **kw: Any
641) -> Union[int, None]:
642    """
643    Return the on-disk size of the pipe's target table in bytes.
644
645    Parameters
646    ----------
647    debug: bool, default False
648        Verbosity toggle.
649
650    Returns
651    -------
652    An `int` of the number of bytes occupied by the pipe's target table,
653    or `None` if the size could not be determined (e.g. the connector does
654    not implement `get_pipe_size()` or the table does not exist).
655    """
656    from meerschaum.utils.warnings import warn
657    from meerschaum.utils.venv import Venv
658    from meerschaum.connectors import get_connector_plugin
659    from meerschaum.utils.misc import filter_keywords
660
661    connector = self.instance_connector
662    try:
663        with Venv(get_connector_plugin(connector)):
664            if not hasattr(connector, 'get_pipe_size'):
665                return None
666            kwargs = filter_keywords(
667                connector.get_pipe_size,
668                debug=debug,
669                **kw
670            )
671            return connector.get_pipe_size(self, **kwargs)
672    except NotImplementedError:
673        return None
674    except Exception as e:
675        warn(f"Failed to get the size of {self}:\n{e}", stack=False)
676    return None

Return the on-disk size of the pipe's target table in bytes.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
  • An int of the number of bytes occupied by the pipe's target table,
  • or None if the size could not be determined (e.g. the connector does
  • not implement get_pipe_size() or the table does not exist).
def get_doc(self, **kwargs) -> Optional[Dict[str, Any]]:
1060def get_doc(self, **kwargs) -> Union[Dict[str, Any], None]:
1061    """
1062    Convenience function to return a single row as a dictionary (or `None`) from `Pipe.get_data().
1063    Keywords arguments are passed to `Pipe.get_data()`.
1064    """
1065    from meerschaum.utils.warnings import warn
1066    kwargs['limit'] = 1
1067    kwargs['as_docs'] = True
1068    try:
1069        docs = self.get_data(**kwargs)
1070        if not docs:
1071            return None
1072        return docs[0]
1073    except Exception as e:
1074        warn(f"Failed to read value from {self}:\n{e}", stack=False)
1075        return None

Convenience function to return a single row as a dictionary (or None) from Pipe.get_data(). Keywords arguments are passed toPipe.get_data()`.

def get_docs(self, **kwargs) -> list[dict[str, typing.Any]]:
1077def get_docs(self, **kwargs) -> list[dict[str, Any]]:
1078    """
1079    Convenience method to return a pipe's data as a list of dictionaries.
1080    Relies on `get_pipe_docs` from the instance connector if implemented.
1081    """
1082    kwargs['as_docs'] = True
1083    return self.get_data(**kwargs)

Convenience method to return a pipe's data as a list of dictionaries. Relies on get_pipe_docs from the instance connector if implemented.

def get_value( self, column: str, params: Optional[Dict[str, Any]] = None, **kwargs: Any) -> Any:
1085def get_value(
1086    self,
1087    column: str,
1088    params: Optional[Dict[str, Any]] = None,
1089    **kwargs: Any
1090) -> Any:
1091    """
1092    Convenience function to return a single value (or `None`) from `Pipe.get_data()`.
1093    Keywords arguments are passed to `Pipe.get_data()`.
1094    """
1095    from meerschaum.utils.warnings import warn
1096    kwargs['select_columns'] = [column]
1097    kwargs['limit'] = 1
1098    kwargs['as_docs'] = True
1099    try:
1100        docs = self.get_data(params=params, **kwargs)
1101        if not docs:
1102            return None
1103        if column not in docs[0]:
1104            raise ValueError(f"Column '{column}' was not included in the result set.")
1105        return docs[0][column]
1106    except Exception as e:
1107        warn(f"Failed to read value from {self}:\n{e}", stack=False)
1108        return None

Convenience function to return a single value (or None) from Pipe.get_data(). Keywords arguments are passed to Pipe.get_data().

def get_chunk_interval( self, chunk_interval: Union[datetime.timedelta, int, NoneType] = None, debug: bool = False) -> Union[datetime.timedelta, int]:
679def get_chunk_interval(
680    self,
681    chunk_interval: Union[timedelta, int, None] = None,
682    debug: bool = False,
683) -> Union[timedelta, int]:
684    """
685    Get the chunk interval to use for this pipe.
686
687    The size is read from the `verify` parameters. Any one of these aliased keys may be used
688    (the first present, in this priority order, wins):
689
690        - `verify.chunk_minutes` (the default; 43200 — 30 days — if none is set)
691        - `verify.chunk_hours`
692        - `verify.chunk_days`
693        - `verify.chunk_weeks`
694        - `verify.chunk_years`
695        - `verify.chunk_seconds`
696
697    For an integer datetime axis, `verify.chunk_range` (if set) is used verbatim as the chunk size
698    in epoch units. Otherwise the time-based size above is converted to epoch units via the pipe's
699    `precision`, or — preserving legacy behavior when no `precision` is set — its minutes are used
700    verbatim.
701
702    Parameters
703    ----------
704    chunk_interval: Union[timedelta, int, None], default None
705        If provided, use this value instead of the `verify` keys.
706        An `int` is already in the axis's own units — epoch units for an integer axis,
707        minutes otherwise — and is used verbatim. A `timedelta` is a duration, so on an
708        integer axis it is scaled into epoch units by the pipe's `precision`.
709
710    Returns
711    -------
712    The chunk interval (`timedelta` or `int`) to use with this pipe's `datetime` axis.
713    """
714    dt_col = self.columns.get('datetime', None)
715    dt_dtype = self.dtypes.get(dt_col, 'datetime') if dt_col is not None else 'datetime'
716    is_int_axis = 'int' in str(dt_dtype).lower()
717    verify_params = self.parameters.get('verify', {})
718
719    ### An explicit `chunk_interval` argument overrides the `verify` keys.
720    if chunk_interval is not None:
721        if dt_col is not None and is_int_axis:
722            ### An `int` is already in epoch units; a duration must be scaled like the
723            ### no-argument path below.
724            if isinstance(chunk_interval, int):
725                return chunk_interval
726            return self._scale_delta_to_int_axis(chunk_interval)
727
728        chunk_minutes = (
729            chunk_interval
730            if isinstance(chunk_interval, int)
731            else int(chunk_interval.total_seconds() / 60)
732        )
733        return timedelta(minutes=chunk_minutes)
734
735    ### Integer axis: an explicit `verify.chunk_range` is the chunk size in epoch units, verbatim.
736    if dt_col is not None and is_int_axis:
737        chunk_range = verify_params.get('chunk_range', None)
738        if chunk_range is not None:
739            return int(chunk_range)
740
741    ### Resolve the time-based chunk size from the aliased `verify.chunk_*` keys (priority order
742    ### matches the `bound_*` aliases). Falls back to the configured `chunk_minutes` default.
743    chunk_delta = None
744    for suffix in ('minutes', 'hours', 'days', 'weeks', 'years', 'seconds'):
745        val = verify_params.get('chunk_' + suffix, None)
746        if val is None:
747            continue
748        ### `timedelta` has no `years` kwarg; approximate a year as 365 days.
749        chunk_delta = timedelta(days=(val * 365)) if suffix == 'years' else timedelta(**{suffix: val})
750        break
751    if chunk_delta is None:
752        default_chunk_minutes = get_config('pipes', 'parameters', 'verify', 'chunk_minutes')
753        chunk_delta = timedelta(minutes=default_chunk_minutes)
754
755    if dt_col is None:
756        return chunk_delta
757
758    if is_int_axis:
759        return self._scale_delta_to_int_axis(chunk_delta)
760
761    return chunk_delta

Get the chunk interval to use for this pipe.

The size is read from the verify parameters. Any one of these aliased keys may be used (the first present, in this priority order, wins):

- `verify.chunk_minutes` (the default; 43200 — 30 days — if none is set)
- `verify.chunk_hours`
- `verify.chunk_days`
- `verify.chunk_weeks`
- `verify.chunk_years`
- `verify.chunk_seconds`

For an integer datetime axis, verify.chunk_range (if set) is used verbatim as the chunk size in epoch units. Otherwise the time-based size above is converted to epoch units via the pipe's precision, or — preserving legacy behavior when no precision is set — its minutes are used verbatim.

Parameters
  • chunk_interval (Union[timedelta, int, None], default None): If provided, use this value instead of the verify keys. An int is already in the axis's own units — epoch units for an integer axis, minutes otherwise — and is used verbatim. A timedelta is a duration, so on an integer axis it is scaled into epoch units by the pipe's precision.
Returns
  • The chunk interval (timedelta or int) to use with this pipe's datetime axis.
def get_chunk_bounds( self, begin: Union[datetime.datetime, int, NoneType] = None, end: Union[datetime.datetime, int, NoneType] = None, bounded: bool = False, chunk_interval: Union[datetime.timedelta, int, NoneType] = None, align: bool = False, debug: bool = False) -> List[Tuple[Union[datetime.datetime, int, NoneType], Union[datetime.datetime, int, NoneType]]]:
785def get_chunk_bounds(
786    self,
787    begin: Union[datetime, int, None] = None,
788    end: Union[datetime, int, None] = None,
789    bounded: bool = False,
790    chunk_interval: Union[timedelta, int, None] = None,
791    align: bool = False,
792    debug: bool = False,
793) -> List[
794    Tuple[
795        Union[datetime, int, None],
796        Union[datetime, int, None],
797    ]
798]:
799    """
800    Return a list of datetime bounds for iterating over the pipe's `datetime` axis.
801
802    Parameters
803    ----------
804    begin: Union[datetime, int, None], default None
805        If provided, do not select less than this value.
806        Otherwise the first chunk will be unbounded.
807
808    end: Union[datetime, int, None], default None
809        If provided, do not select greater than or equal to this value.
810        Otherwise the last chunk will be unbounded.
811
812    bounded: bool, default False
813        If `True`, do not include `None` in the first chunk.
814
815    chunk_interval: Union[timedelta, int, None], default None
816        If provided, use this interval for the size of chunk boundaries.
817        The default value for this pipe may be set
818        under `pipe.parameters['verify']['chunk_minutes']`.
819
820    align: bool, default False
821        If `True`, anchor the interior chunk boundaries to a fixed Unix-epoch grid (the same
822        grid used for native range partitioning) rather than to `begin`. This makes the
823        boundaries deterministic across re-syncs and aligned with the pipe's partitions
824        (used by `Pipe.verify()`). The first chunk's lower bound and the last chunk's upper
825        bound are still clamped to `begin` / `end`.
826
827    debug: bool, default False
828        Verbosity toggle.
829
830    Returns
831    -------
832    A list of chunk bounds (datetimes or integers).
833    If unbounded, the first and last chunks will include `None`.
834    """
835    from datetime import timedelta
836    from meerschaum.utils.dtypes import are_dtypes_equal
837    from meerschaum.utils.misc import interval_str
838    include_less_than_begin = not bounded and begin is None
839    include_greater_than_end = not bounded and end is None
840    if begin is None:
841        begin = self.get_sync_time(newest=False, debug=debug)
842    consolidate_end_chunk = False
843    if end is None:
844        end = self.get_sync_time(newest=True, debug=debug)
845        if end is not None and hasattr(end, 'tzinfo'):
846            end += timedelta(minutes=1)
847            consolidate_end_chunk = True
848        elif are_dtypes_equal(str(type(end)), 'int'):
849            end += 1
850            consolidate_end_chunk = True
851
852    if begin is None and end is None:
853        return [(None, None)]
854
855    begin, end = self.parse_date_bounds(begin, end, debug=debug)
856
857    if begin and end:
858        if begin >= end:
859            return (
860                [(begin, begin)]
861                if bounded
862                else [(begin, None)]
863            )
864        if end <= begin:
865            return (
866                [(end, end)]
867                if bounded
868                else [(None, begin)]
869            )
870
871    ### Set the chunk interval under `pipe.parameters['verify']['chunk_minutes']`.
872    chunk_interval = self.get_chunk_interval(chunk_interval, debug=debug)
873
874    ### Anchor the interior boundaries to a fixed Unix-epoch grid (matching native range
875    ### partitioning, see `SQLConnector._partition_bounds`) so chunk edges line up with partition
876    ### edges and stay deterministic regardless of `begin`. The first chunk is clamped back to
877    ### `begin` below.
878    begin_cursor = begin
879    if align and begin is not None:
880        if isinstance(chunk_interval, int):
881            begin_cursor = (int(begin) // chunk_interval) * chunk_interval
882        else:
883            epoch = (
884                datetime(1970, 1, 1, tzinfo=begin.tzinfo)
885                if getattr(begin, 'tzinfo', None) is not None
886                else datetime(1970, 1, 1)
887            )
888            n = (begin - epoch) // chunk_interval
889            begin_cursor = epoch + (n * chunk_interval)
890
891    ### Build a list of tuples containing the chunk boundaries
892    ### so that we can sync multiple chunks in parallel.
893    ### Run `verify pipes --workers 1` to sync chunks in series.
894    chunk_bounds = []
895    num_chunks = 0
896    max_chunks = 1_000_000
897    while begin_cursor < end:
898        end_cursor = begin_cursor + chunk_interval
899        chunk_bounds.append((begin_cursor, end_cursor))
900        begin_cursor = end_cursor
901        num_chunks += 1
902        if num_chunks >= max_chunks:
903            raise ValueError(
904                f"Too many chunks of size '{interval_str(chunk_interval)}' "
905                f"between '{begin}' and '{end}'."
906            )
907
908    if num_chunks > 1 and consolidate_end_chunk:
909        last_bounds, second_last_bounds = chunk_bounds[-1], chunk_bounds[-2]
910        chunk_bounds = chunk_bounds[:-2]
911        chunk_bounds.append((second_last_bounds[0], last_bounds[1]))
912
913    ### The chunk interval might be too large.
914    if not chunk_bounds and end >= begin:
915        chunk_bounds = [(begin, end)]
916
917    ### Truncate the last chunk to the end timestamp.
918    if chunk_bounds[-1][1] > end:
919        chunk_bounds[-1] = (chunk_bounds[-1][0], end)
920
921    ### Pop the last chunk if its bounds are equal.
922    if chunk_bounds[-1][0] == chunk_bounds[-1][1]:
923        chunk_bounds = chunk_bounds[:-1]
924
925    ### Clamp the epoch-aligned first chunk's lower bound back to the requested `begin` so the
926    ### returned range still starts exactly at `begin` (only the interior edges are grid-aligned).
927    if (
928        align
929        and chunk_bounds
930        and chunk_bounds[0][0] is not None
931        and chunk_bounds[0][0] < begin
932    ):
933        chunk_bounds[0] = (begin, chunk_bounds[0][1])
934
935    if include_less_than_begin:
936        chunk_bounds = [(None, begin)] + chunk_bounds
937    if include_greater_than_end:
938        chunk_bounds = chunk_bounds + [(end, None)]
939
940    return chunk_bounds

Return a list of datetime bounds for iterating over the pipe's datetime axis.

Parameters
  • begin (Union[datetime, int, None], default None): If provided, do not select less than this value. Otherwise the first chunk will be unbounded.
  • end (Union[datetime, int, None], default None): If provided, do not select greater than or equal to this value. Otherwise the last chunk will be unbounded.
  • bounded (bool, default False): If True, do not include None in the first chunk.
  • chunk_interval (Union[timedelta, int, None], default None): If provided, use this interval for the size of chunk boundaries. The default value for this pipe may be set under pipe.parameters['verify']['chunk_minutes'].
  • align (bool, default False): If True, anchor the interior chunk boundaries to a fixed Unix-epoch grid (the same grid used for native range partitioning) rather than to begin. This makes the boundaries deterministic across re-syncs and aligned with the pipe's partitions (used by Pipe.verify()). The first chunk's lower bound and the last chunk's upper bound are still clamped to begin / end.
  • debug (bool, default False): Verbosity toggle.
Returns
  • A list of chunk bounds (datetimes or integers).
  • If unbounded, the first and last chunks will include None.
def get_chunk_bounds_batches( self, chunk_bounds: List[Tuple[Union[datetime.datetime, int, NoneType], Union[datetime.datetime, int, NoneType]]], batchsize: Optional[int] = None, workers: Optional[int] = None, debug: bool = False) -> List[Tuple[Tuple[Union[datetime.datetime, int, NoneType], Union[datetime.datetime, int, NoneType]], ...]]:
943def get_chunk_bounds_batches(
944    self,
945    chunk_bounds: List[Tuple[Union[datetime, int, None], Union[datetime, int, None]]],
946    batchsize: Optional[int] = None,
947    workers: Optional[int] = None,
948    debug: bool = False,
949) -> List[
950    Tuple[
951        Tuple[
952            Union[datetime, int, None],
953            Union[datetime, int, None],
954        ], ...
955    ]
956]:
957    """
958    Return a list of tuples of chunk bounds of size `batchsize`.
959
960    Parameters
961    ----------
962    chunk_bounds: List[Tuple[Union[datetime, int, None], Union[datetime, int, None]]]
963        A list of chunk_bounds (see `Pipe.get_chunk_bounds()`).
964
965    batchsize: Optional[int], default None
966        How many chunks to include in a batch. Defaults to `Pipe.get_num_workers()`.
967
968    workers: Optional[int], default None
969        If `batchsize` is `None`, use this as the desired number of workers.
970        Passed to `Pipe.get_num_workers()`.
971
972    Returns
973    -------
974    A list of tuples of chunk bound tuples.
975    """
976    from meerschaum.utils.misc import iterate_chunks
977    
978    if batchsize is None:
979        batchsize = self.get_num_workers(workers=workers)
980
981    return [
982        tuple(
983            _batch_chunk_bounds
984            for _batch_chunk_bounds in batch
985            if _batch_chunk_bounds is not None
986        )
987        for batch in iterate_chunks(chunk_bounds, batchsize)
988        if batch
989    ]

Return a list of tuples of chunk bounds of size batchsize.

Parameters
  • chunk_bounds (List[Tuple[Union[datetime, int, None], Union[datetime, int, None]]]): A list of chunk_bounds (see Pipe.get_chunk_bounds()).
  • batchsize (Optional[int], default None): How many chunks to include in a batch. Defaults to Pipe.get_num_workers().
  • workers (Optional[int], default None): If batchsize is None, use this as the desired number of workers. Passed to Pipe.get_num_workers().
Returns
  • A list of tuples of chunk bound tuples.
def parse_date_bounds( self, *dt_vals: Union[datetime.datetime, int, NoneType], debug: bool = False) -> Union[datetime.datetime, int, str, NoneType, Tuple[Union[datetime.datetime, int, str, NoneType]]]:
 992def parse_date_bounds(self, *dt_vals: Union[datetime, int, None], debug: bool = False) -> Union[
 993    datetime,
 994    int,
 995    str,
 996    None,
 997    Tuple[Union[datetime, int, str, None]]
 998]:
 999    """
1000    Given a date bound (begin, end), coerce a timezone if necessary.
1001    """
1002    from meerschaum.utils.misc import is_int
1003    from meerschaum.utils.dtypes import coerce_timezone, MRSM_PD_DTYPES, are_dtypes_equal
1004    from meerschaum.utils.warnings import warn
1005    dateutil_parser = mrsm.attempt_import('dateutil.parser')
1006
1007    _columns = None
1008    _dtypes = None
1009
1010    def _get_coercion_info():
1011        nonlocal _columns, _dtypes
1012        if _columns is None:
1013            _columns = self.get_parameters(debug=debug).get('columns', {}) or {}
1014        if _dtypes is None:
1015            _dtypes = self.get_dtypes(debug=debug)
1016
1017    def _parse_date_bound(dt_val):
1018        if dt_val is None:
1019            return None
1020
1021        if isinstance(dt_val, int):
1022            return dt_val
1023
1024        if dt_val == '':
1025            return ''
1026
1027        if is_int(dt_val):
1028            return int(dt_val)
1029
1030        if isinstance(dt_val, str):
1031            try:
1032                dt_val = dateutil_parser.parse(dt_val)
1033            except Exception as e:
1034                warn(f"Could not parse '{dt_val}' as datetime:\n{e}")
1035                return None
1036
1037        _get_coercion_info()
1038        dt_col = _columns.get('datetime', None)
1039        dt_typ = str(_dtypes.get(dt_col, 'datetime'))
1040        if are_dtypes_equal(dt_typ, 'int'):
1041            if self.get_parameters(debug=debug).get('precision'):
1042                from meerschaum.utils.dtypes import datetime_to_int
1043                return datetime_to_int(dt_val, self.precision['unit'])
1044            from meerschaum.utils.warnings import error
1045            error(
1046                f"Cannot use datetime bound '{dt_val}' on the non-epoch integer axis "
1047                f"of {self}.\n    Pass an integer instead, or set the `precision` parameter.",
1048                ValueError,
1049            )
1050        if dt_typ == 'datetime':
1051            dt_typ = MRSM_PD_DTYPES['datetime']
1052        return coerce_timezone(dt_val, strip_utc=('utc' not in dt_typ.lower()))
1053
1054    bounds = tuple(_parse_date_bound(dt_val) for dt_val in dt_vals)
1055    if len(bounds) == 1:
1056        return bounds[0]
1057    return bounds

Given a date bound (begin, end), coerce a timezone if necessary.

def register(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
12def register(
13    self,
14    debug: bool = False,
15    **kw: Any
16) -> SuccessTuple:
17    """
18    Register a new Pipe along with its attributes.
19
20    Parameters
21    ----------
22    debug: bool, default False
23        Verbosity toggle.
24
25    kw: Any
26        Keyword arguments to pass to `instance_connector.register_pipe()`.
27
28    Returns
29    -------
30    A `SuccessTuple` of success, message.
31    """
32    if self.temporary:
33        return False, "Cannot register pipes created with `temporary=True` (read-only)."
34
35    from meerschaum.utils.formatting import get_console
36    from meerschaum.utils.venv import Venv
37    from meerschaum.connectors import get_connector_plugin, custom_types
38    from meerschaum.config._patch import apply_patch_to_config
39
40    import warnings
41    with warnings.catch_warnings():
42        warnings.simplefilter('ignore')
43        try:
44            _conn = self.connector
45        except Exception:
46            _conn = None
47
48        if isinstance(_conn, str):
49            _conn = None
50
51    if (
52        _conn is not None
53        and
54        (_conn.type == 'plugin' or _conn.type in custom_types)
55        and
56        getattr(_conn, 'register', None) is not None
57    ):
58        try:
59            with Venv(get_connector_plugin(_conn), debug=debug):
60                params = self.connector.register(self)
61        except Exception:
62            get_console().print_exception()
63            params = None
64        params = {} if params is None else params
65        if not isinstance(params, dict):
66            from meerschaum.utils.warnings import warn
67            warn(
68                f"Invalid parameters returned from `register()` in connector {self.connector}:\n"
69                + f"{params}"
70            )
71        else:
72            self.parameters = apply_patch_to_config(params, self.parameters)
73
74    if not self.parameters:
75        cols = self.columns if self.columns else {'datetime': None, 'id': None}
76        self.parameters = {
77            'columns': cols,
78        }
79
80    ### Refuse an index column which cannot be sorted or deduplicated, rather than
81    ### registering a pipe whose second sync is guaranteed to fail.
82    indices_success, indices_msg = self.validate_indices(debug=debug)
83    if not indices_success:
84        return indices_success, indices_msg
85
86    with Venv(get_connector_plugin(self.instance_connector)):
87        return self.instance_connector.register_pipe(self, debug=debug, **kw)

Register a new Pipe along with its attributes.

Parameters
  • debug (bool, default False): Verbosity toggle.
  • kw (Any): Keyword arguments to pass to instance_connector.register_pipe().
Returns
attributes: Dict[str, Any]
20@property
21def attributes(self) -> Dict[str, Any]:
22    """
23    Return a dictionary of a pipe's keys and parameters.
24    These values are reflected directly from the pipes table of the instance.
25    """
26    from meerschaum.config import get_config
27    from meerschaum.config._patch import apply_patch_to_config
28    from meerschaum.utils.venv import Venv
29    from meerschaum.connectors import get_connector_plugin
30    from meerschaum.utils.dtypes import get_current_timestamp
31
32    if self.temporary:
33        return self._attributes
34
35    with self._attributes_lock:
36        # Instance connectors may read local parameters while fetching metadata.
37        if self.__dict__.get('_attributes_loading', False):
38            return self._attributes
39        timeout_seconds = get_config('pipes', 'attributes', 'local_cache_timeout_seconds')
40        now = get_current_timestamp('ms', as_int=True) / 1000
41        sync_time = self._get_cached_value('_attributes_sync_time', debug=self.debug)
42        timed_out = (
43            sync_time is None
44            or (timeout_seconds is not None and (now - sync_time) >= timeout_seconds)
45        )
46        if timed_out:
47            local_attributes = self._get_cached_value('attributes', debug=self.debug) or {}
48            self._attributes_loading = True
49            try:
50                with Venv(get_connector_plugin(self.instance_connector)):
51                    instance_attributes = self.instance_connector.get_pipe_attributes(self)
52                if not isinstance(instance_attributes, dict) or (
53                    not instance_attributes and self._get_cached_value('_id') is not None
54                ):
55                    raise RuntimeError(f"Missing metadata for registered pipe {self}.")
56                self._cache_value(
57                    'attributes',
58                    apply_patch_to_config(instance_attributes, local_attributes),
59                    memory_only=True,
60                    debug=self.debug,
61                )
62            except Exception as e:
63                self._attributes_load_failures = self.__dict__.get('_attributes_load_failures', 0) + 1
64                self._clear_cache_key('_attributes_sync_time', debug=self.debug)
65                warn(f"Failed to load attributes for {self}: {e}")
66                return self._attributes
67            finally:
68                self._attributes_loading = False
69            self._cache_value(
70                '_attributes_sync_time', get_current_timestamp('ms', as_int=True) / 1000,
71                memory_only=True, debug=self.debug,
72            )
73
74        return self._attributes

Return a dictionary of a pipe's keys and parameters. These values are reflected directly from the pipes table of the instance.

parameters: Optional[Dict[str, Any]]
202@property
203def parameters(self) -> Optional[Dict[str, Any]]:
204    """
205    Return the parameters dictionary of the pipe.
206    """
207    return self.get_parameters(debug=self.debug)

Return the parameters dictionary of the pipe.

columns: Optional[Dict[str, str]]
219@property
220def columns(self) -> Union[Dict[str, str], None]:
221    """
222    Return the `columns` dictionary defined in `meerschaum.Pipe.parameters`.
223    """
224    cols = self.parameters.get('columns', {})
225    if not isinstance(cols, dict):
226        return {}
227    return {col_ix: col for col_ix, col in cols.items() if col and col_ix}

Return the columns dictionary defined in meerschaum.Pipe.parameters.

indices: Optional[Dict[str, Union[str, List[str]]]]
244@property
245def indices(self) -> Union[Dict[str, Union[str, List[str]]], None]:
246    """
247    Return the `indices` dictionary defined in `meerschaum.Pipe.parameters`.
248    """
249    _parameters = self.get_parameters(debug=self.debug)
250    indices_key = (
251        'indexes'
252        if 'indexes' in _parameters
253        else 'indices'
254    )
255
256    _indices = _parameters.get(indices_key, {})
257    _columns = self.columns
258    dt_col = _columns.get('datetime', None)
259    if not isinstance(_indices, dict):
260        _indices = {}
261    unique_cols = list(set((
262        [dt_col]
263        if dt_col
264        else []
265    ) + [
266        col
267        for col_ix, col in _columns.items()
268        if col and col_ix != 'datetime'
269    ]))
270    return {
271        **({'unique': unique_cols} if len(unique_cols) > 1 else {}),
272        **{col_ix: col for col_ix, col in _columns.items() if col},
273        **_indices
274    }

Return the indices dictionary defined in meerschaum.Pipe.parameters.

indexes: Optional[Dict[str, Union[str, List[str]]]]
277@property
278def indexes(self) -> Union[Dict[str, Union[str, List[str]]], None]:
279    """
280    Alias for `meerschaum.Pipe.indices`.
281    """
282    return self.indices
dtypes: Dict[str, Any]
333@property
334def dtypes(self) -> Dict[str, Any]:
335    """
336    If defined, return the `dtypes` dictionary defined in `meerschaum.Pipe.parameters`.
337    """
338    return self.get_dtypes(refresh=False, debug=self.debug)

If defined, return the dtypes dictionary defined in meerschaum.Pipe.parameters.

autoincrement: bool
491@property
492def autoincrement(self) -> bool:
493    """
494    Return the `autoincrement` parameter for the pipe.
495    """
496    return self.parameters.get('autoincrement', False)

Return the autoincrement parameter for the pipe.

autotime: bool
507@property
508def autotime(self) -> bool:
509    """
510    Return the `autotime` parameter for the pipe.
511    """
512    return self.parameters.get('autotime', False)

Return the autotime parameter for the pipe.

upsert: bool
458@property
459def upsert(self) -> bool:
460    """
461    Return whether `upsert` is set for the pipe.
462    """
463    return self.parameters.get('upsert', False)

Return whether upsert is set for the pipe.

static: bool
474@property
475def static(self) -> bool:
476    """
477    Return whether `static` is set for the pipe.
478    """
479    return self.parameters.get('static', False)

Return whether static is set for the pipe.

tzinfo: Optional[datetime.timezone]
523@property
524def tzinfo(self) -> Union[None, timezone]:
525    """
526    Return `timezone.utc` if the pipe is timezone-aware.
527    """
528    _tzinfo = self._get_cached_value('tzinfo', debug=self.debug)
529    if _tzinfo is not None:
530        return _tzinfo if _tzinfo != 'None' else None
531
532    _tzinfo = None
533    dt_col = self.columns.get('datetime', None)
534    dt_typ = str(self.dtypes.get(dt_col, 'datetime')) if dt_col else None
535    if self.autotime:
536        ts_col = mrsm.get_config('pipes', 'autotime', 'column_name_if_datetime_missing')
537        ts_typ = self.dtypes.get(ts_col, 'datetime')
538        dt_typ = ts_typ
539
540    if dt_typ and 'utc' in dt_typ.lower() or dt_typ == 'datetime':
541        _tzinfo = timezone.utc
542
543    self._cache_value('tzinfo', (_tzinfo if _tzinfo is not None else 'None'), debug=self.debug)
544    return _tzinfo

Return timezone.utc if the pipe is timezone-aware.

enforce: bool
547@property
548def enforce(self) -> bool:
549    """
550    Return the `enforce` parameter for the pipe.
551    """
552    return self.parameters.get('enforce', True)

Return the enforce parameter for the pipe.

null_indices: bool
563@property
564def null_indices(self) -> bool:
565    """
566    Return the `null_indices` parameter for the pipe.
567    """
568    return self.parameters.get('null_indices', True)

Return the null_indices parameter for the pipe.

mixed_numerics: bool
579@property
580def mixed_numerics(self) -> bool:
581    """
582    Return the `mixed_numerics` parameter for the pipe.
583    """
584    return self.parameters.get('mixed_numerics', True)

Return the mixed_numerics parameter for the pipe.

def get_columns(self, *args: str, error: bool = False) -> Union[str, Tuple[str]]:
595def get_columns(self, *args: str, error: bool = False) -> Union[str, Tuple[str]]:
596    """
597    Check if the requested columns are defined.
598
599    Parameters
600    ----------
601    *args: str
602        The column names to be retrieved.
603
604    error: bool, default False
605        If `True`, raise an `Exception` if the specified column is not defined.
606
607    Returns
608    -------
609    A tuple of the same size of `args` or a `str` if `args` is a single argument.
610
611    Examples
612    --------
613    >>> pipe = mrsm.Pipe('test', 'test')
614    >>> pipe.columns = {'datetime': 'dt', 'id': 'id'}
615    >>> pipe.get_columns('datetime', 'id')
616    ('dt', 'id')
617    >>> pipe.get_columns('value', error=True)
618    Exception:  🛑 Missing 'value' column for Pipe('test', 'test').
619    """
620    from meerschaum.utils.warnings import error as _error
621    if not args:
622        args = tuple(self.columns.keys())
623    col_names = []
624    for col in args:
625        col_name = None
626        try:
627            col_name = self.columns[col]
628            if col_name is None and error:
629                _error(f"Please define the name of the '{col}' column for {self}.")
630        except Exception:
631            col_name = None
632        if col_name is None and error:
633            _error(f"Missing '{col}'" + f" column for {self}.")
634        col_names.append(col_name)
635    if len(col_names) == 1:
636        return col_names[0]
637    return tuple(col_names)

Check if the requested columns are defined.

Parameters
  • *args (str): The column names to be retrieved.
  • error (bool, default False): If True, raise an Exception if the specified column is not defined.
Returns
  • A tuple of the same size of args or a str if args is a single argument.
Examples
>>> pipe = mrsm.Pipe('test', 'test')
>>> pipe.columns = {'datetime': 'dt', 'id': 'id'}
>>> pipe.get_columns('datetime', 'id')
('dt', 'id')
>>> pipe.get_columns('value', error=True)
Exception:  🛑 Missing 'value' column for Pipe('test', 'test').
def get_columns_types( self, refresh: bool = False, debug: bool = False) -> Optional[Dict[str, str]]:
640def get_columns_types(
641    self,
642    refresh: bool = False,
643    debug: bool = False,
644) -> Union[Dict[str, str], None]:
645    """
646    Get a dictionary of a pipe's column names and their types.
647
648    Parameters
649    ----------
650    refresh: bool, default False
651        If `True`, invalidate the cache and fetch directly from the instance connector.
652
653    debug: bool, default False:
654        Verbosity toggle.
655
656    Returns
657    -------
658    A dictionary of column names (`str`) to column types (`str`).
659
660    Examples
661    --------
662    >>> pipe.get_columns_types()
663    {
664      'dt': 'TIMESTAMP WITH TIMEZONE',
665      'id': 'BIGINT',
666      'val': 'DOUBLE PRECISION',
667    }
668    >>>
669    """
670    from meerschaum.connectors import get_connector_plugin
671    from meerschaum.utils.dtypes import get_current_timestamp
672
673    now = get_current_timestamp('ms', as_int=True) / 1000
674    cache_seconds = (
675        mrsm.get_config('pipes', 'static', 'static_schema_cache_seconds')
676        if self.static
677        else mrsm.get_config('pipes', 'dtypes', 'columns_types_cache_seconds')
678    )
679    if refresh:
680        self._clear_cache_key('_columns_types_timestamp', debug=debug)
681        self._clear_cache_key('_columns_types', debug=debug)
682
683    _columns_types = self._get_cached_value('_columns_types', debug=debug)
684    if _columns_types:
685        columns_types_timestamp = self._get_cached_value('_columns_types_timestamp', debug=debug)
686        if columns_types_timestamp is not None:
687            delta = now - columns_types_timestamp
688            if delta < cache_seconds:
689                if debug:
690                    dprint(
691                        f"Returning cached `columns_types` for {self} "
692                        f"({round(delta, 2)} seconds old)."
693                    )
694                return _columns_types
695
696    with mrsm.Venv(get_connector_plugin(self.instance_connector)):
697        _columns_types = (
698            self.instance_connector.get_pipe_columns_types(self, debug=debug)
699            if hasattr(self.instance_connector, 'get_pipe_columns_types')
700            else None
701        )
702
703    self._cache_value('_columns_types', _columns_types, debug=debug)
704    self._cache_value('_columns_types_timestamp', now, debug=debug)
705    return _columns_types or {}

Get a dictionary of a pipe's column names and their types.

Parameters
  • refresh (bool, default False): If True, invalidate the cache and fetch directly from the instance connector.
  • debug (bool, default False:): Verbosity toggle.
Returns
  • A dictionary of column names (str) to column types (str).
Examples
>>> pipe.get_columns_types()
{
  'dt': 'TIMESTAMP WITH TIMEZONE',
  'id': 'BIGINT',
  'val': 'DOUBLE PRECISION',
}
>>>
def get_columns_indices( self, debug: bool = False, refresh: bool = False) -> Dict[str, List[Dict[str, str]]]:
708def get_columns_indices(
709    self,
710    debug: bool = False,
711    refresh: bool = False,
712) -> Dict[str, List[Dict[str, str]]]:
713    """
714    Return a dictionary mapping columns to index information.
715    """
716    from meerschaum.connectors import get_connector_plugin
717    from meerschaum.utils.dtypes import get_current_timestamp
718
719    now = get_current_timestamp('ms', as_int=True) / 1000
720    cache_seconds = (
721        mrsm.get_config('pipes', 'static', 'static_schema_cache_seconds')
722        if self.static
723        else mrsm.get_config('pipes', 'dtypes', 'columns_types_cache_seconds')
724    )
725    if refresh:
726        self._clear_cache_key('_columns_indices_timestamp', debug=debug)
727        self._clear_cache_key('_columns_indices', debug=debug)
728
729    _columns_indices = self._get_cached_value('_columns_indices', debug=debug)
730
731    if _columns_indices:
732        columns_indices_timestamp = self._get_cached_value('_columns_indices_timestamp', debug=debug)
733        if columns_indices_timestamp is not None:
734            delta = now - columns_indices_timestamp
735            if delta < cache_seconds:
736                if debug:
737                    dprint(
738                        f"Returning cached `columns_indices` for {self} "
739                        f"({round(delta, 2)} seconds old)."
740                    )
741                return _columns_indices
742
743    with mrsm.Venv(get_connector_plugin(self.instance_connector)):
744        _columns_indices = (
745            self.instance_connector.get_pipe_columns_indices(self, debug=debug)
746            if hasattr(self.instance_connector, 'get_pipe_columns_indices')
747            else None
748        )
749
750    self._cache_value('_columns_indices', _columns_indices, debug=debug)
751    self._cache_value('_columns_indices_timestamp', now, debug=debug)
752    return {k: v for k, v in _columns_indices.items() if k and v} or {}

Return a dictionary mapping columns to index information.

def get_indices(self) -> Dict[str, str]:
1164def get_indices(self) -> Dict[str, str]:
1165    """
1166    Return a dictionary mapping index keys to their names in the database.
1167
1168    Returns
1169    -------
1170    A dictionary of index keys to index names.
1171    """
1172    from meerschaum.connectors import get_connector_plugin
1173    with mrsm.Venv(get_connector_plugin(self.instance_connector)):
1174        if hasattr(self.instance_connector, 'get_pipe_index_names'):
1175            result = self.instance_connector.get_pipe_index_names(self)
1176        else:
1177            result = {}
1178    
1179    return result

Return a dictionary mapping index keys to their names in the database.

Returns
  • A dictionary of index keys to index names.
def get_parameters( self, apply_symlinks: bool = True, refresh: bool = False, debug: bool = False, _visited: Optional[set[Pipe]] = None) -> Dict[str, Any]:
 79def get_parameters(
 80    self,
 81    apply_symlinks: bool = True,
 82    refresh: bool = False,
 83    debug: bool = False,
 84    _visited: 'Optional[set[mrsm.Pipe]]' = None,
 85) -> Dict[str, Any]:
 86    """
 87    Return the `parameters` dictionary of the pipe.
 88
 89    Parameters
 90    ----------
 91    apply_symlinks: bool, default True
 92        If `True`, resolve references to parameters from other pipes.
 93
 94    refresh: bool, default False
 95        If `True`, pull the latest attributes for the pipe.
 96
 97    Returns
 98    -------
 99    The pipe's parameters dictionary.
100    """
101    from copy import deepcopy
102    from meerschaum.config._patch import apply_patch_to_config
103    from meerschaum.config._read_config import search_and_substitute_config
104
105    is_top_level = _visited is None
106    if _visited is None:
107        _visited = {self}
108
109    if refresh:
110        _ = self._invalidate_cache(hard=True)
111        with self._attributes_lock:
112            self._clear_cache_key('_attributes_sync_time', debug=debug)
113            self.__dict__.pop('_target', None)
114        ### Drop any memoized resolution so a later non-refresh call recomputes from fresh state.
115        _ = self.__dict__.pop('_resolved_parameters_raw', None)
116        _ = self.__dict__.pop('_resolved_parameters', None)
117        _ = self.__dict__.pop('_resolved_parameters_symlinks', None)
118
119    raw_parameters = self.attributes.get('parameters', {})
120    if not apply_symlinks:
121        return raw_parameters
122
123    ### Resolving references + `{{ Pipe() }}` / `MRSM{}` symlinks is pure-Python but expensive
124    ### (it walks reference pipes and may build connectors), and `get_parameters` is a hot path
125    ### hit by `.dtypes`, `.columns`, `.precision`, etc. Memoize the resolved result, keyed on the
126    ### identity of the raw parameters dict: every mutation path (`update_parameters`, the setter,
127    ### `edit`) reassigns `_attributes['parameters']` to a new object, so identity changing is a
128    ### reliable invalidation signal. Schema is *not* part of this — dynamic-schema freshness is
129    ### handled separately by `get_columns_types`' TTL cache, so this is safe for dynamic pipes.
130    ### Only memoize the top-level entry (not nested reference resolution, which threads `_visited`
131    ### for cycle detection) and only the default symlink-resolving, non-refreshing call.
132    can_memoize = is_top_level and not refresh
133    if can_memoize and self.__dict__.get('_resolved_parameters_raw', None) is raw_parameters:
134        self._symlinks = self.__dict__.get('_resolved_parameters_symlinks', {})
135        ### Return a copy so callers that mutate the result (e.g. `infer_dtypes(persist=True)`)
136        ### don't corrupt the memo.
137        return deepcopy(self.__dict__['_resolved_parameters'])
138
139    parameters = {}
140    for ref_pipe in self.references:
141        try:
142            if ref_pipe in _visited:
143                warn(f"Circular reference detected in {self}: chain involves {ref_pipe}.")
144                return search_and_substitute_config(raw_parameters)
145
146            _visited.add(ref_pipe)
147            if refresh:
148                _ = _cached_base_params.pop(ref_pipe, None)
149            base_params = _cached_base_params.get(ref_pipe, None)
150            if base_params is None:
151                base_params = ref_pipe.get_parameters(
152                    apply_symlinks=apply_symlinks,
153                    _visited=_visited,
154                    debug=debug,
155                )
156                _cached_base_params[ref_pipe] = base_params
157                if debug:
158                    dprint(f"base_params from {ref_pipe} for {self}:")
159                    mrsm.pprint(base_params)
160            else:
161                if debug:
162                    dprint(f"Using cached base_params from {ref_pipe} for {self}")
163        except Exception as e:
164            warn(f"Failed to resolve reference pipe for {self}: {e}")
165            base_params = {}
166
167        parameters = apply_patch_to_config(parameters, base_params)
168
169    parameters = apply_patch_to_config(parameters, raw_parameters)
170
171    from meerschaum.utils.pipes import replace_pipes_syntax
172    self._symlinks = {}
173
174    def recursive_replace(obj: Any, path: tuple) -> Any:
175        if isinstance(obj, dict):
176            return {k: recursive_replace(v, path + (k,)) for k, v in obj.items()}
177        if isinstance(obj, list):
178            return [recursive_replace(elem, path + (i,)) for i, elem in enumerate(obj)]
179        if isinstance(obj, str):
180            substituted_val = replace_pipes_syntax(obj, _pipe=self)
181            if substituted_val != obj:
182                self._symlinks[path] = {
183                    'original': obj,
184                    'substituted': substituted_val,
185                }
186            return substituted_val
187        return obj
188
189    resolved_parameters = search_and_substitute_config(recursive_replace(parameters, tuple()))
190
191    if can_memoize:
192        ### Hold a reference to the raw dict so its identity can't be reused by a freed object,
193        ### and stash the symlinks captured above alongside the resolved result.
194        self.__dict__['_resolved_parameters_raw'] = raw_parameters
195        self.__dict__['_resolved_parameters'] = resolved_parameters
196        self.__dict__['_resolved_parameters_symlinks'] = self._symlinks
197        return deepcopy(resolved_parameters)
198
199    return resolved_parameters

Return the parameters dictionary of the pipe.

Parameters
  • apply_symlinks (bool, default True): If True, resolve references to parameters from other pipes.
  • refresh (bool, default False): If True, pull the latest attributes for the pipe.
Returns
  • The pipe's parameters dictionary.
def get_dtypes( self, infer: bool = True, refresh: bool = False, debug: bool = False) -> Dict[str, Any]:
352def get_dtypes(
353    self,
354    infer: bool = True,
355    refresh: bool = False,
356    debug: bool = False,
357) -> Dict[str, Any]:
358    """
359    If defined, return the `dtypes` dictionary defined in `meerschaum.Pipe.parameters`.
360
361    Parameters
362    ----------
363    infer: bool, default True
364        If `True`, include the implicit existing dtypes.
365        Else only return the explicitly configured dtypes (e.g. `Pipe.parameters['dtypes']`).
366
367    refresh: bool, default False
368        If `True`, invalidate any cache and return the latest known dtypes.
369
370    Returns
371    -------
372    A dictionary mapping column names to dtypes.
373    """
374    from meerschaum.config._patch import apply_patch_to_config
375    from meerschaum.utils.dtypes import MRSM_ALIAS_DTYPES
376    parameters = self.get_parameters(refresh=refresh, debug=debug)
377    configured_dtypes = parameters.get('dtypes', {})
378    if debug:
379        dprint(f"Configured dtypes for {self}:")
380        mrsm.pprint(configured_dtypes)
381
382    remote_dtypes = (
383        self.infer_dtypes(persist=False, refresh=refresh, debug=debug)
384        if infer
385        else {}
386    )
387    if debug and infer:
388        dprint(f"Remote dtypes for {self}:")
389        mrsm.pprint(remote_dtypes)
390
391    patched_dtypes = apply_patch_to_config((remote_dtypes or {}), (configured_dtypes or {}))
392
393    dt_col = parameters.get('columns', {}).get('datetime', None)
394    primary_col = parameters.get('columns', {}).get('primary', None)
395    _dtypes = {
396        col: MRSM_ALIAS_DTYPES.get(typ, typ)
397        for col, typ in patched_dtypes.items()
398        if col and typ
399    }
400    if dt_col and dt_col not in configured_dtypes:
401        _dtypes[dt_col] = 'datetime'
402    if primary_col and parameters.get('autoincrement', False) and primary_col not in _dtypes:
403        _dtypes[primary_col] = 'int'
404
405    return _dtypes

If defined, return the dtypes dictionary defined in meerschaum.Pipe.parameters.

Parameters
  • infer (bool, default True): If True, include the implicit existing dtypes. Else only return the explicitly configured dtypes (e.g. Pipe.parameters['dtypes']).
  • refresh (bool, default False): If True, invalidate any cache and return the latest known dtypes.
Returns
  • A dictionary mapping column names to dtypes.
def update_parameters( self, parameters_patch: Dict[str, Any], persist: bool = True, debug: bool = False) -> Tuple[bool, str]:
1182def update_parameters(
1183    self,
1184    parameters_patch: Dict[str, Any],
1185    persist: bool = True,
1186    debug: bool = False,
1187) -> mrsm.SuccessTuple:
1188    """
1189    Apply a patch to a pipe's `parameters` dictionary.
1190
1191    Parameters
1192    ----------
1193    parameters_patch: Dict[str, Any]
1194        The patch to be applied to `Pipe.parameters`.
1195
1196    persist: bool, default True
1197        If `True`, call `Pipe.edit()` to persist the new parameters.
1198    """
1199    from meerschaum.config import apply_patch_to_config
1200    if 'parameters' not in self._attributes:
1201        self._attributes['parameters'] = {}
1202
1203    self._attributes['parameters'] = apply_patch_to_config(
1204        self._attributes['parameters'],
1205        parameters_patch,
1206    )
1207
1208    if self.temporary:
1209        persist = False
1210
1211    if not persist:
1212        return True, "Success"
1213
1214    return self.edit(debug=debug)

Apply a patch to a pipe's parameters dictionary.

Parameters
  • parameters_patch (Dict[str, Any]): The patch to be applied to Pipe.parameters.
  • persist (bool, default True): If True, call Pipe.edit() to persist the new parameters.
def validate_indices(self, debug: bool = False) -> Tuple[bool, str]:
412def validate_indices(self, debug: bool = False) -> mrsm.SuccessTuple:
413    """
414    Return whether this pipe's index columns can actually be used as indices.
415
416    A `json` column holds `dict` or `list` values. Syncing sorts and deduplicates
417    on the index columns in Pandas (`Series.unique()`, `DataFrame.sort_values()`),
418    and both raise `TypeError: unhashable type: 'dict'` on container values. The
419    failure surfaces on the *second* sync — the first one has nothing to diff
420    against — so it is worth reporting up front instead of at that point.
421
422    Note this is a client-side limitation, not a database one: on PostgreSQL the
423    `json` dtype is stored as `JSONB`, which does support B-tree, hash, and GIN
424    indices. Non-unique performance indices declared under `indices` are pure
425    DDL and are not affected; only the index *roles* in `columns` are.
426
427    Returns
428    -------
429    A `SuccessTuple` of whether the index columns are valid.
430    """
431    columns = self.columns or {}
432    if not columns:
433        return True, "Success"
434
435    dtypes = self.get_dtypes(infer=False, debug=debug)
436    offenders = {
437        col_ix: col
438        for col_ix, col in columns.items()
439        if col and str(dtypes.get(col, '')).lower() in UNHASHABLE_DTYPES
440    }
441    if not offenders:
442        return True, "Success"
443
444    listing = ', '.join(
445        f"'{col}' (as the '{col_ix}' index)"
446        for col_ix, col in sorted(offenders.items())
447    )
448    return False, (
449        f"Cannot use a 'json' column as an index column: {listing}.\n"
450        "    JSON values are dictionaries or lists, which cannot be sorted or "
451        "deduplicated when syncing.\n"
452        "    Either give the column a scalar dtype (e.g. 'str'), or move it out of "
453        "`columns` and\n"
454        "    declare it under `indices` if you only need a database index on it."
455    )

Return whether this pipe's index columns can actually be used as indices.

A json column holds dict or list values. Syncing sorts and deduplicates on the index columns in Pandas (Series.unique(), DataFrame.sort_values()), and both raise TypeError: unhashable type: 'dict' on container values. The failure surfaces on the second sync — the first one has nothing to diff against — so it is worth reporting up front instead of at that point.

Note this is a client-side limitation, not a database one: on PostgreSQL the json dtype is stored as JSONB, which does support B-tree, hash, and GIN indices. Non-unique performance indices declared under indices are pure DDL and are not affected; only the index roles in columns are.

Returns
tags: Optional[List[str]]
310@property
311def tags(self) -> Union[List[str], None]:
312    """
313    If defined, return the `tags` list defined in `meerschaum.Pipe.parameters`.
314    """
315    return self.parameters.get('tags', [])

If defined, return the tags list defined in meerschaum.Pipe.parameters.

def get_id(self, **kw: Any) -> Union[int, str, NoneType]:
755def get_id(self, **kw: Any) -> Union[int, str, None]:
756    """
757    Fetch a pipe's ID from its instance connector.
758    If the pipe is not registered, return `None`.
759    """
760    if self.temporary:
761        return None
762
763    from meerschaum.utils.venv import Venv
764    from meerschaum.connectors import get_connector_plugin
765
766    with Venv(get_connector_plugin(self.instance_connector)):
767        if hasattr(self.instance_connector, 'get_pipe_id'):
768            return self.instance_connector.get_pipe_id(self, **kw)
769
770    return None

Fetch a pipe's ID from its instance connector. If the pipe is not registered, return None.

id: Union[int, str, uuid.UUID, NoneType]
773@property
774def id(self) -> Union[int, str, uuid.UUID, None]:
775    """
776    Fetch and cache a pipe's ID.
777    """
778    _id = self._get_cached_value('_id', debug=self.debug)
779    if _id is None:
780        _id = self.get_id(debug=self.debug)
781        if _id is not None:
782            self._cache_value('_id', _id, debug=self.debug)
783    return _id

Fetch and cache a pipe's ID.

def get_val_column(self, debug: bool = False) -> Optional[str]:
786def get_val_column(self, debug: bool = False) -> Union[str, None]:
787    """
788    Return the name of the value column if it's defined, otherwise make an educated guess.
789    If not set in the `columns` dictionary, return the first numeric column that is not
790    an ID or datetime column.
791    If none may be found, return `None`.
792
793    Parameters
794    ----------
795    debug: bool, default False:
796        Verbosity toggle.
797
798    Returns
799    -------
800    Either a string or `None`.
801    """
802    if debug:
803        dprint('Attempting to determine the value column...')
804    try:
805        val_name = self.get_columns('value')
806    except Exception:
807        val_name = None
808    if val_name is not None:
809        if debug:
810            dprint(f"Value column: {val_name}")
811        return val_name
812
813    cols = self.columns
814    if cols is None:
815        if debug:
816            dprint('No columns could be determined. Returning...')
817        return None
818    try:
819        dt_name = self.get_columns('datetime', error=False)
820    except Exception:
821        dt_name = None
822    try:
823        id_name = self.get_columns('id', errors=False)
824    except Exception:
825        id_name = None
826
827    if debug:
828        dprint(f"dt_name: {dt_name}")
829        dprint(f"id_name: {id_name}")
830
831    cols_types = self.get_columns_types(debug=debug)
832    if cols_types is None:
833        return None
834    if debug:
835        dprint(f"cols_types: {cols_types}")
836    if dt_name is not None:
837        cols_types.pop(dt_name, None)
838    if id_name is not None:
839        cols_types.pop(id_name, None)
840
841    candidates = []
842    candidate_keywords = {'float', 'double', 'precision', 'int', 'numeric',}
843    for search_term in candidate_keywords:
844        for col, typ in cols_types.items():
845            if search_term in typ.lower():
846                candidates.append(col)
847                break
848    if not candidates:
849        if debug:
850            dprint("No value column could be determined.")
851        return None
852
853    return candidates[0]

Return the name of the value column if it's defined, otherwise make an educated guess. If not set in the columns dictionary, return the first numeric column that is not an ID or datetime column. If none may be found, return None.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
  • Either a string or None.
parents: List[Pipe]
856@property
857def parents(self) -> List[mrsm.Pipe]:
858    """
859    Return a list of `meerschaum.Pipe` objects to be designated as parents.
860    """
861    _cached_parents = self.__dict__.get('_parents', None)
862    if _cached_parents is not None:
863        return _cached_parents
864
865    from meerschaum.utils.pipes import get_pipe_from_value
866    base_params = self.get_parameters()
867    key = 'parents' if 'parents' in base_params else 'parent'
868    parents_refs = base_params.get(key, None) or []
869    if isinstance(parents_refs, str) or isinstance(parents_refs, dict):
870        parents_refs = [parents_refs]
871
872    if not parents_refs:
873        return []
874
875    self._parents = [get_pipe_from_value(val, _pipe=self) for val in parents_refs]
876    return self._parents

Return a list of meerschaum.Pipe objects to be designated as parents.

parent: Optional[Pipe]
879@property
880def parent(self) -> Union[mrsm.Pipe, None]:
881    """
882    Return the first pipe in `self.parents` or `None`.
883    """
884    _parents = self.parents
885    if not _parents:
886        return None
887
888    return _parents[0]

Return the first pipe in self.parents or None.

children: List[Pipe]
924@property
925def children(self) -> List[mrsm.Pipe]:
926    """
927    Return a list of `meerschaum.Pipe` objects to be designated as children.
928    """
929    _cached_children = self.__dict__.get('_children', None)
930    if _cached_children is not None:
931        return _cached_children
932
933    from meerschaum.utils.pipes import get_pipe_from_value
934    base_params = self.get_parameters()
935    key = 'children' if 'children' in base_params else 'child'
936    children_refs = base_params.get(key, None) or []
937    if isinstance(children_refs, str) or isinstance(children_refs, dict):
938        children_refs = [children_refs]
939
940    if not children_refs:
941        return []
942
943    self._children = [get_pipe_from_value(val, _pipe=self) for val in children_refs]
944    return self._children

Return a list of meerschaum.Pipe objects to be designated as children.

child: Pipe | None
947@property
948def child(self) -> mrsm.Pipe | None:
949    """
950    Return the first pipe in `self.children` or None.
951    """
952    _children = self.children
953    if not _children:
954        return None
955
956    return _children[0]

Return the first pipe in self.children or None.

reference: Pipe | None
1016@property
1017def reference(self) -> mrsm.Pipe | None:
1018    """
1019    Return the first pipe in `self.references` or None.
1020    """
1021    _references = self.references
1022    if not _references:
1023        return None
1024
1025    return _references[0]

Return the first pipe in self.references or None.

references: List[Pipe]
 993@property
 994def references(self) -> List[mrsm.Pipe]:
 995    """
 996    Return a list of `meerschaum.Pipe` objects to be designated as references.
 997    """
 998    _cached_references = self.__dict__.get('_references', None)
 999    if _cached_references is not None:
1000        return _cached_references
1001
1002    from meerschaum.utils.pipes import get_pipe_from_value
1003    base_params = self.get_parameters(apply_symlinks=False)
1004    key = 'references' if 'references' in base_params else 'reference'
1005    refs = base_params.get(key, None) or []
1006    if isinstance(refs, str) or isinstance(refs, dict):
1007        refs = [refs]
1008
1009    if not refs:
1010        return []
1011
1012    self._references = [get_pipe_from_value(val, _pipe=self) for val in refs]
1013    return self._references

Return a list of meerschaum.Pipe objects to be designated as references.

target: str
1063@property
1064def target(self) -> str:
1065    """
1066    The target table name.
1067    Registered pipes do not memoize guessed defaults while metadata failures are outstanding.
1068    You can set the target name under on of the following keys
1069    (checked in this order):
1070      - `target`
1071      - `target_name`
1072      - `target_table`
1073      - `target_table_name`
1074    """
1075    cached_target = self.__dict__.get('_target', None)
1076    if cached_target:
1077        return cached_target
1078
1079    load_failures = self.__dict__.get('_attributes_load_failures', 0)
1080    params = self.parameters
1081    target_val = params.get('target', None)
1082    if target_val:
1083        self.__dict__['_target'] = target_val
1084        return target_val
1085
1086    default_target = self._target_legacy()
1087    default_targets = {default_target}
1088    potential_keys = ('target_name', 'target_table', 'target_table_name')
1089    _target = None
1090    for k in potential_keys:
1091        if k in params:
1092            _target = params[k]
1093            break
1094
1095    _target = _target or default_target
1096
1097    if self.instance_connector.type == 'sql':
1098        from meerschaum.utils.sql import truncate_item_name
1099        truncated_target = truncate_item_name(_target, self.instance_connector.flavor)
1100        default_targets.add(truncated_target)
1101        warned_target = self.__dict__.get('_warned_target', False)
1102        if truncated_target != _target and not warned_target:
1103            if self.instance_connector.flavor not in ('oracle', 'mysql', 'mariadb'):
1104                warn(
1105                    f"The target '{_target}' is too long for '{self.instance_connector.flavor}', "
1106                    + f"will use {truncated_target} instead."
1107                )
1108            self.__dict__['_warned_target'] = True
1109            _target = truncated_target
1110
1111    if _target not in default_targets:
1112        self.target = _target
1113
1114    # Parameter resolution may retry a failed load; its first snapshot is still stale.
1115    resolved_cleanly = load_failures == self.__dict__.get('_attributes_load_failures', 0)
1116    if (params and resolved_cleanly) or self.temporary or self.id is None:
1117        self.__dict__['_target'] = _target
1118    return _target

The target table name. Registered pipes do not memoize guessed defaults while metadata failures are outstanding. You can set the target name under on of the following keys (checked in this order):

  • target
  • target_name
  • target_table
  • target_table_name
def guess_datetime(self) -> Optional[str]:
1142def guess_datetime(self) -> Union[str, None]:
1143    """
1144    Try to determine a pipe's datetime column.
1145    """
1146    _dtypes = self.dtypes
1147
1148    ### Abort if the user explictly disallows a datetime index.
1149    if 'datetime' in _dtypes:
1150        if _dtypes['datetime'] is None:
1151            return None
1152
1153    from meerschaum.utils.dtypes import are_dtypes_equal
1154    dt_cols = [
1155        col
1156        for col, typ in _dtypes.items()
1157        if are_dtypes_equal(typ, 'datetime')
1158    ]
1159    if not dt_cols:
1160        return None
1161    return dt_cols[0]

Try to determine a pipe's datetime column.

precision: Dict[str, Union[str, int]]
1306@property
1307def precision(self) -> Dict[str, Union[str, int]]:
1308    """
1309    Return the configured or detected precision.
1310    """
1311    return self.get_precision(debug=self.debug)

Return the configured or detected precision.

def get_precision(self, debug: bool = False) -> Dict[str, Union[str, int]]:
1217def get_precision(self, debug: bool = False) -> Dict[str, Union[str, int]]:
1218    """
1219    Return the timestamp precision unit and interval for the `datetime` axis.
1220    """
1221    from meerschaum.utils.dtypes import (
1222        MRSM_PRECISION_UNITS_SCALARS,
1223        MRSM_PRECISION_UNITS_ALIASES,
1224        MRSM_PD_DTYPES,
1225        are_dtypes_equal,
1226    )
1227    from meerschaum._internal.static import STATIC_CONFIG
1228
1229    _precision = self._get_cached_value('precision', debug=debug)
1230    if _precision:
1231        if debug:
1232            dprint(f"Returning cached precision: {_precision}")
1233        return _precision
1234
1235    parameters = self.parameters
1236    _precision = parameters.get('precision', {})
1237    if isinstance(_precision, str):
1238        _precision = {'unit': _precision}
1239    default_precision_unit = STATIC_CONFIG['dtypes']['datetime']['default_precision_unit']
1240
1241    if not _precision:
1242
1243        dt_col = parameters.get('columns', {}).get('datetime', None)
1244        if not dt_col and self.autotime:
1245            dt_col = mrsm.get_config('pipes', 'autotime', 'column_name_if_datetime_missing')
1246        if not dt_col:
1247            if debug:
1248                dprint(f"No datetime axis, returning default precision '{default_precision_unit}'.")
1249            return {'unit': default_precision_unit}
1250
1251        dt_typ = self.dtypes.get(dt_col, 'datetime')
1252        if are_dtypes_equal(dt_typ, 'datetime'):
1253            if dt_typ == 'datetime':
1254                dt_typ = MRSM_PD_DTYPES['datetime']
1255                if debug:
1256                    dprint(f"Datetime type is `datetime`, assuming {dt_typ} precision.")
1257
1258            _precision = {
1259                'unit': (
1260                    dt_typ
1261                    .split('[', maxsplit=1)[-1]
1262                    .split(',', maxsplit=1)[0]
1263                    .split(' ', maxsplit=1)[0]
1264                ).rstrip(']')
1265            }
1266
1267            if debug:
1268                dprint(f"Extracted precision '{_precision['unit']}' from type '{dt_typ}'.")
1269
1270        elif are_dtypes_equal(dt_typ, 'int'):
1271            _precision = {
1272                'unit': (
1273                    'second'
1274                    if '32' in dt_typ
1275                    else default_precision_unit
1276                )
1277            }
1278        elif are_dtypes_equal(dt_typ, 'date'):
1279            if debug:
1280                dprint("Datetime axis is 'date', falling back to 'day' precision.")
1281            _precision = {'unit': 'day'}
1282
1283    precision_unit = _precision.get('unit', default_precision_unit)
1284    precision_interval = _precision.get('interval', None)
1285    true_precision_unit = MRSM_PRECISION_UNITS_ALIASES.get(precision_unit, precision_unit)
1286    if true_precision_unit is None:
1287        if debug:
1288            dprint(f"No precision could be determined, falling back to '{default_precision_unit}'.")
1289        true_precision_unit = default_precision_unit
1290
1291    if true_precision_unit not in MRSM_PRECISION_UNITS_SCALARS:
1292        from meerschaum.utils.misc import items_str
1293        raise ValueError(
1294            f"Invalid precision unit '{true_precision_unit}'.\n"
1295            "Accepted values are "
1296            f"{items_str(list(MRSM_PRECISION_UNITS_SCALARS) + list(MRSM_PRECISION_UNITS_ALIASES))}."
1297        )
1298
1299    _precision = {'unit': true_precision_unit}
1300    if precision_interval:
1301        _precision['interval'] = precision_interval
1302    self._cache_value('precision', _precision, debug=debug)
1303    return self._precision

Return the timestamp precision unit and interval for the datetime axis.

def show( self, nopretty: bool = False, debug: bool = False, **kw) -> Tuple[bool, str]:
12def show(
13    self,
14    nopretty: bool = False,
15    debug: bool = False,
16    **kw
17) -> SuccessTuple:
18    """
19    Show attributes of a Pipe.
20
21    Parameters
22    ----------
23    nopretty: bool, default False
24        If `True`, simply print the JSON of the pipe's attributes.
25
26    debug: bool, default False
27        Verbosity toggle.
28
29    Returns
30    -------
31    A `SuccessTuple` of success, message.
32
33    """
34    import json
35    from meerschaum.utils.formatting import (
36        pprint, make_header, ANSI, highlight_pipes, fill_ansi, get_console,
37    )
38    from meerschaum.utils.packages import import_rich, attempt_import
39    from meerschaum.utils.warnings import info
40    attributes_json = json.dumps(self.attributes)
41    if not nopretty:
42        _to_print = f"Attributes for {self}:"
43        if ANSI:
44            _to_print = fill_ansi(highlight_pipes(make_header(_to_print)), 'magenta')
45            print(_to_print)
46            rich = import_rich()
47            rich_json = attempt_import('rich.json')
48            get_console().print(rich_json.JSON(attributes_json))
49        else:
50            print(_to_print)
51    else:
52        print(attributes_json)
53
54    return True, "Success"

Show attributes of a Pipe.

Parameters
  • nopretty (bool, default False): If True, simply print the JSON of the pipe's attributes.
  • debug (bool, default False): Verbosity toggle.
Returns
def edit( self, patch: bool = False, interactive: bool = False, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
 21def edit(
 22    self,
 23    patch: bool = False,
 24    interactive: bool = False,
 25    debug: bool = False,
 26    **kw: Any
 27) -> SuccessTuple:
 28    """
 29    Edit a Pipe's configuration.
 30
 31    Parameters
 32    ----------
 33    patch: bool, default False
 34        If `patch` is True, update parameters by cascading rather than overwriting.
 35    interactive: bool, default False
 36        If `True`, open an editor for the user to make changes to the pipe's YAML file.
 37    debug: bool, default False
 38        Verbosity toggle.
 39
 40    Returns
 41    -------
 42    A `SuccessTuple` of success, message.
 43
 44    """
 45    from meerschaum.utils.venv import Venv
 46    from meerschaum.connectors import get_connector_plugin
 47
 48    if self.temporary:
 49        return False, "Cannot edit pipes created with `temporary=True` (read-only)."
 50
 51    self._invalidate_cache(hard=True, debug=debug)
 52
 53    if hasattr(self, '_symlinks'):
 54        from meerschaum.utils.misc import get_val_from_dict_path, set_val_in_dict_path
 55        for path, vals in self._symlinks.items():
 56            current_val = get_val_from_dict_path(self.parameters, path)
 57            if current_val == vals['substituted']:
 58                set_val_in_dict_path(self.parameters, path, vals['original'])
 59
 60    if not interactive:
 61        ### Refuse an index column which cannot be sorted or deduplicated.
 62        indices_success, indices_msg = self.validate_indices(debug=debug)
 63        if not indices_success:
 64            return indices_success, indices_msg
 65
 66        with Venv(get_connector_plugin(self.instance_connector)):
 67            return self.instance_connector.edit_pipe(self, patch=patch, debug=debug, **kw)
 68
 69    import meerschaum.config.paths as paths
 70    from meerschaum.utils.misc import edit_file
 71    parameters_filename = str(self) + '.yaml'
 72    parameters_path = paths.PIPES_CACHE_RESOURCES_PATH / parameters_filename
 73
 74    from meerschaum.utils.yaml import yaml
 75
 76    edit_text = f"Edit the parameters for {self}"
 77    edit_top = '#' * (len(edit_text) + 4)
 78    edit_header = edit_top + f'\n# {edit_text} #\n' + edit_top + '\n\n'
 79
 80    from meerschaum.config import get_config
 81    parameters = dict(get_config('pipes', 'parameters', patch=True))
 82    from meerschaum.config._patch import apply_patch_to_config
 83    raw_parameters = self.attributes.get('parameters', {})
 84    parameters = apply_patch_to_config(parameters, raw_parameters)
 85
 86    ### write parameters to yaml file
 87    with open(parameters_path, 'w+') as f:
 88        f.write(edit_header)
 89        yaml.dump(parameters, stream=f, sort_keys=False)
 90
 91    ### only quit editing if yaml is valid
 92    editing = True
 93    while editing:
 94        edit_file(parameters_path)
 95        try:
 96            with open(parameters_path, 'r') as f:
 97                file_parameters = yaml.load(f.read())
 98        except Exception as e:
 99            from meerschaum.utils.warnings import warn
100            warn(f"Invalid format defined for '{self}':\n\n{e}")
101            input(f"Press [Enter] to correct the configuration for '{self}': ")
102        else:
103            editing = False
104
105    self.parameters = file_parameters
106
107    if debug:
108        from meerschaum.utils.formatting import pprint
109        pprint(self.parameters)
110
111    with Venv(get_connector_plugin(self.instance_connector)):
112        return self.instance_connector.edit_pipe(self, patch=patch, debug=debug, **kw)

Edit a Pipe's configuration.

Parameters
  • patch (bool, default False): If patch is True, update parameters by cascading rather than overwriting.
  • interactive (bool, default False): If True, open an editor for the user to make changes to the pipe's YAML file.
  • debug (bool, default False): Verbosity toggle.
Returns
def edit_definition( self, yes: bool = False, noask: bool = False, force: bool = False, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
115def edit_definition(
116    self,
117    yes: bool = False,
118    noask: bool = False,
119    force: bool = False,
120    debug : bool = False,
121    **kw : Any
122) -> SuccessTuple:
123    """
124    Edit a pipe's definition file and update its configuration.
125    **NOTE:** This function is interactive and should not be used in automated scripts!
126
127    Returns
128    -------
129    A `SuccessTuple` of success, message.
130
131    """
132    if self.temporary:
133        return False, "Cannot edit pipes created with `temporary=True` (read-only)."
134
135    from meerschaum.connectors import instance_types
136    if (self.connector is None or isinstance(self.connector, str)) or self.connector.type not in instance_types:
137        return self.edit(interactive=True, debug=debug, **kw)
138
139    import json
140    from meerschaum.utils.warnings import info, warn
141    from meerschaum.utils.debug import dprint
142    from meerschaum.config._patch import apply_patch_to_config
143    from meerschaum.utils.misc import edit_file
144
145    _parameters = self.parameters
146    if 'fetch' not in _parameters:
147        _parameters['fetch'] = {}
148
149    def _edit_api():
150        from meerschaum.utils.prompt import prompt, yes_no
151        info(
152            f"Please enter the keys of the source pipe from '{self.connector}'.\n" +
153            "Type 'None' for None, or empty when there is no default. Press [CTRL+C] to skip."
154        )
155
156        _keys = { 'connector_keys' : None, 'metric_key' : None, 'location_key' : None }
157        for k in _keys:
158            _keys[k] = _parameters['fetch'].get(k, None)
159
160        for k, v in _keys.items():
161            try:
162                _keys[k] = prompt(k.capitalize().replace('_', ' ') + ':', icon=True, default=v)
163            except KeyboardInterrupt:
164                continue
165            if _keys[k] in ('', 'None', '\'None\'', '[None]'):
166                _keys[k] = None
167
168        _parameters['fetch'] = apply_patch_to_config(_parameters['fetch'], _keys)
169
170        info("You may optionally specify additional filter parameters as JSON.")
171        print("  Parameters are translated into a 'WHERE x AND y' clause, and lists are IN clauses.")
172        print("  For example, the following JSON would correspond to 'WHERE x = 1 AND y IN (2, 3)':")
173        print(json.dumps({'x': 1, 'y': [2, 3]}, indent=2, separators=(',', ': ')))
174        if force or yes_no(
175            "Would you like to add additional filter parameters?",
176            yes=yes, noask=noask
177        ):
178            import meerschaum.config.paths as paths
179            definition_filename = str(self) + '.json'
180            definition_path = paths.PIPES_CACHE_RESOURCES_PATH / definition_filename
181            try:
182                definition_path.touch()
183                with open(definition_path, 'w+') as f:
184                    json.dump(_parameters.get('fetch', {}).get('params', {}), f, indent=2)
185            except Exception as e:
186                return False, f"Failed writing file '{definition_path}':\n" + str(e)
187
188            _params = None
189            while True:
190                edit_file(definition_path)
191                try:
192                    with open(definition_path, 'r') as f:
193                        _params = json.load(f)
194                except Exception as e:
195                    warn(f'Failed to read parameters JSON:\n{e}', stack=False)
196                    if force or yes_no(
197                        "Would you like to try again?\n  "
198                        + "If not, the parameters JSON file will be ignored.",
199                        noask=noask, yes=yes
200                    ):
201                        continue
202                    _params = None
203                break
204            if _params is not None:
205                if 'fetch' not in _parameters:
206                    _parameters['fetch'] = {}
207                _parameters['fetch']['params'] = _params
208
209        self.parameters = _parameters
210        return True, "Success"
211
212    def _edit_sql():
213        import textwrap
214        import meerschaum.config.paths as paths
215        from meerschaum.utils.misc import edit_file
216        definition_filename = str(self) + '.sql'
217        definition_path = paths.PIPES_CACHE_RESOURCES_PATH / definition_filename
218
219        sql_definition = _parameters['fetch'].get('definition', None)
220        if sql_definition is None:
221            sql_definition = ''
222        sql_definition = textwrap.dedent(sql_definition).lstrip()
223
224        try:
225            definition_path.touch()
226            with open(definition_path, 'w+') as f:
227                f.write(sql_definition)
228        except Exception as e:
229            return False, f"Failed writing file '{definition_path}':\n" + str(e)
230
231        edit_file(definition_path)
232        try:
233            with open(definition_path, 'r', encoding='utf-8') as f:
234                file_definition = f.read()
235        except Exception as e:
236            return False, f"Failed reading file '{definition_path}':\n" + str(e)
237
238        if sql_definition == file_definition:
239            return False, f"No changes made to definition for {self}."
240
241        if ' ' not in file_definition:
242            return False, f"Invalid SQL definition for {self}."
243
244        if debug:
245            dprint("Read SQL definition:\n\n" + file_definition)
246        _parameters['fetch']['definition'] = file_definition
247        self.parameters = _parameters
248        return True, "Success"
249
250    locals()['_edit_' + str(self.connector.type)]()
251    return self.edit(interactive=False, debug=debug, **kw)

Edit a pipe's definition file and update its configuration. NOTE: This function is interactive and should not be used in automated scripts!

Returns
def update(self, *args, **kw) -> Tuple[bool, str]:
13def update(self, *args, **kw) -> SuccessTuple:
14    """
15    Update a pipe's parameters in its instance.
16    """
17    kw['interactive'] = False
18    return self.edit(*args, **kw)

Update a pipe's parameters in its instance.

def sync( self, df: Union[pandas.DataFrame, Dict[str, List[Any]], List[Dict[str, Any]], str, meerschaum.core.Pipe._sync.InferFetch] = <class 'meerschaum.core.Pipe._sync.InferFetch'>, begin: Union[datetime.datetime, int, str, NoneType] = '', end: Union[datetime.datetime, int, NoneType] = None, force: bool = False, retries: int = 10, min_seconds: int = 1, check_existing: bool = True, enforce_dtypes: bool = True, blocking: bool = True, workers: Optional[int] = None, callback: Optional[Callable[[Tuple[bool, str]], Any]] = None, error_callback: Optional[Callable[[Exception], Any]] = None, chunksize: Optional[int] = -1, sync_chunks: bool = True, debug: bool = False, _inplace: bool = True, **kw: Any) -> Tuple[bool, str]:
 41def sync(
 42    self,
 43    df: Union[
 44        pd.DataFrame,
 45        Dict[str, List[Any]],
 46        List[Dict[str, Any]],
 47        str,
 48        InferFetch
 49    ] = InferFetch,
 50    begin: Union[datetime, int, str, None] = '',
 51    end: Union[datetime, int, None] = None,
 52    force: bool = False,
 53    retries: int = 10,
 54    min_seconds: int = 1,
 55    check_existing: bool = True,
 56    enforce_dtypes: bool = True,
 57    blocking: bool = True,
 58    workers: Optional[int] = None,
 59    callback: Optional[Callable[[Tuple[bool, str]], Any]] = None,
 60    error_callback: Optional[Callable[[Exception], Any]] = None,
 61    chunksize: Optional[int] = -1,
 62    sync_chunks: bool = True,
 63    debug: bool = False,
 64    _inplace: bool = True,
 65    **kw: Any
 66) -> SuccessTuple:
 67    """
 68    Fetch new data from the source and update the pipe's table with new data.
 69    
 70    Get new remote data via fetch, get existing data in the same time period,
 71    and merge the two, only keeping the unseen data.
 72
 73    Parameters
 74    ----------
 75    df: Union[None, pd.DataFrame, Dict[str, List[Any]]], default None
 76        An optional DataFrame to sync into the pipe. Defaults to `None`.
 77        If `df` is a string, it will be parsed via `meerschaum.utils.dataframe.parse_simple_lines()`.
 78
 79    begin: Union[datetime, int, str, None], default ''
 80        Optionally specify the earliest datetime to search for data.
 81
 82    end: Union[datetime, int, str, None], default None
 83        Optionally specify the latest datetime to search for data.
 84
 85    force: bool, default False
 86        If `True`, keep trying to sync untul `retries` attempts.
 87
 88    retries: int, default 10
 89        If `force`, how many attempts to try syncing before declaring failure.
 90
 91    min_seconds: Union[int, float], default 1
 92        If `force`, how many seconds to sleep between retries. Defaults to `1`.
 93
 94    check_existing: bool, default True
 95        If `True`, pull and diff with existing data from the pipe.
 96
 97    enforce_dtypes: bool, default True
 98        If `True`, enforce dtypes on incoming data.
 99        Set this to `False` if the incoming rows are expected to be of the correct dtypes.
100
101    blocking: bool, default True
102        If `True`, wait for sync to finish and return its result, otherwise
103        asyncronously sync (oxymoron?) and return success. Defaults to `True`.
104        Only intended for specific scenarios.
105
106    workers: Optional[int], default None
107        If provided and the instance connector is thread-safe
108        (`pipe.instance_connector.IS_THREAD_SAFE is True`),
109        limit concurrent sync to this many threads.
110
111    callback: Optional[Callable[[Tuple[bool, str]], Any]], default None
112        Callback function which expects a SuccessTuple as input.
113        Only applies when `blocking=False`.
114
115    error_callback: Optional[Callable[[Exception], Any]], default None
116        Callback function which expects an Exception as input.
117        Only applies when `blocking=False`.
118
119    chunksize: int, default -1
120        Specify the number of rows to sync per chunk.
121        If `-1`, resort to system configuration (default is `900`).
122        A `chunksize` of `None` will sync all rows in one transaction.
123
124    sync_chunks: bool, default True
125        If possible, sync chunks while fetching them into memory.
126
127    debug: bool, default False
128        Verbosity toggle. Defaults to False.
129
130    Returns
131    -------
132    A `SuccessTuple` of success (`bool`) and message (`str`).
133    """
134    from meerschaum.utils.debug import dprint, _checkpoint
135    from meerschaum.utils.formatting import get_console
136    from meerschaum.utils.venv import Venv
137    from meerschaum.connectors import get_connector_plugin
138    from meerschaum.utils.misc import df_is_chunk_generator, filter_keywords, filter_arguments
139    from meerschaum.utils.pool import get_pool
140    from meerschaum.config import get_config
141    from meerschaum.utils.dtypes import are_dtypes_equal, get_current_timestamp
142
143    if (callback is not None or error_callback is not None) and blocking:
144        warn("Callback functions are only executed when blocking = False. Ignoring...")
145
146    _checkpoint(_total=2, **kw)
147
148    if chunksize == 0:
149        chunksize = None
150        sync_chunks = False
151
152    begin, end = self.parse_date_bounds(begin, end)
153    kw.update({
154        'begin': begin,
155        'end': end,
156        'force': force,
157        'retries': retries,
158        'min_seconds': min_seconds,
159        'check_existing': check_existing,
160        'blocking': blocking,
161        'workers': workers,
162        'callback': callback,
163        'error_callback': error_callback,
164        'sync_chunks': sync_chunks,
165        'chunksize': chunksize,
166        'safe_copy': True,
167    })
168
169    self._invalidate_cache(debug=debug)
170    self._cache_value('sync_ts', get_current_timestamp('ms'), debug=debug)
171
172    def _sync(
173        p: mrsm.Pipe,
174        df: Union[
175            'pd.DataFrame',
176            Dict[str, List[Any]],
177            List[Dict[str, Any]],
178            str,
179            InferFetch
180        ] = InferFetch,
181    ) -> SuccessTuple:
182        if df is None:
183            p._invalidate_cache(debug=debug)
184            return (
185                False,
186                f"You passed `None` instead of data into `sync()` for {p}.\n"
187                + "Omit the DataFrame to infer fetching.",
188            )
189        ### Ensure that Pipe is registered.
190        if not p.temporary and p.id is None:
191            ### NOTE: This may trigger an interactive session for plugins!
192            register_success, register_msg = p.register(debug=debug)
193            if not register_success:
194                if 'already' not in register_msg:
195                    p._invalidate_cache(debug=debug)
196                    return register_success, register_msg
197
198        ### Checked here as well as in `register()` so that a pipe registered before
199        ### this validation existed reports the reason rather than failing deep in
200        ### Pandas with `unhashable type: 'dict'`.
201        indices_success, indices_msg = p.validate_indices(debug=debug)
202        if not indices_success:
203            p._invalidate_cache(debug=debug)
204            return indices_success, indices_msg
205
206        if isinstance(df, str):
207            from meerschaum.utils.dataframe import parse_simple_lines
208            df = parse_simple_lines(df)
209
210        ### If connector is a plugin with a `sync()` method, return that instead.
211        ### If the plugin does not have a `sync()` method but does have a `fetch()` method,
212        ### use that instead.
213        ### NOTE: The DataFrame must be omitted for the plugin sync method to apply.
214        ### If a DataFrame is provided, continue as expected.
215        if hasattr(df, 'MRSM_INFER_FETCH'):
216            try:
217                if isinstance(p.connector, str):
218                    if ':' not in p.connector_keys:
219                        return True, f"{p} does not support fetching; nothing to do."
220
221                    msg = f"{p} does not have a valid connector."
222                    if p.connector_keys.startswith('plugin:'):
223                        msg += f"\n    Perhaps {p.connector_keys} has a syntax error?"
224                    p._invalidate_cache(debug=debug)
225                    return False, msg
226            except Exception:
227                p._invalidate_cache(debug=debug)
228                return False, f"Unable to create the connector for {p}."
229
230            ### Sync in place if possible.
231            if (
232                str(self.connector) == str(self.instance_connector)
233                and 
234                hasattr(self.instance_connector, 'sync_pipe_inplace')
235                and
236                _inplace
237                and
238                get_config('system', 'experimental', 'inplace_sync')
239            ):
240                with Venv(get_connector_plugin(self.instance_connector)):
241                    p._invalidate_cache(debug=debug)
242                    _args, _kwargs = filter_arguments(
243                        p.instance_connector.sync_pipe_inplace,
244                        p,
245                        debug=debug,
246                        **kw
247                    )
248                    return self.instance_connector.sync_pipe_inplace(
249                        *_args,
250                        **_kwargs
251                    )
252
253            ### Activate and invoke `sync(pipe)` for plugin connectors with `sync` methods.
254            try:
255                if getattr(p.connector, 'sync', None) is not None:
256                    with Venv(get_connector_plugin(p.connector), debug=debug):
257                        _args, _kwargs = filter_arguments(
258                            p.connector.sync,
259                            p,
260                            debug=debug,
261                            **kw
262                        )
263                        return_tuple = p.connector.sync(*_args, **_kwargs)
264                    p._invalidate_cache(debug=debug)
265                    if not isinstance(return_tuple, tuple):
266                        return_tuple = (
267                            False,
268                            f"Plugin '{p.connector.label}' returned non-tuple value: {return_tuple}"
269                        )
270                    return return_tuple
271
272            except Exception as e:
273                get_console().print_exception()
274                msg = f"Failed to sync {p} with exception: '" + str(e) + "'"
275                if debug:
276                    error(msg, silent=False)
277                p._invalidate_cache(debug=debug)
278                return False, msg
279
280            ### Fetch the dataframe from the connector's `fetch()` method.
281            try:
282                with Venv(get_connector_plugin(p.connector), debug=debug):
283                    df = p.fetch(
284                        **filter_keywords(
285                            p.fetch,
286                            debug=debug,
287                            **kw
288                        )
289                    )
290                    kw['safe_copy'] = False
291            except Exception as e:
292                get_console().print_exception(
293                    suppress=[
294                        'meerschaum/core/Pipe/_sync.py',
295                        'meerschaum/core/Pipe/_fetch.py',
296                    ]
297                )
298                msg = f"Failed to fetch data from {p.connector}:\n    {e}"
299                df = None
300
301            if df is None:
302                p._invalidate_cache(debug=debug)
303                return False, f"No data were fetched for {p}."
304
305            if isinstance(df, list):
306                if len(df) == 0:
307                    return True, f"No new rows were returned for {p}."
308
309                ### May be a chunk hook results list.
310                if isinstance(df[0], tuple):
311                    success = all([_success for _success, _ in df])
312                    message = '\n'.join([_message for _, _message in df])
313                    return success, message
314
315            if df is True:
316                p._invalidate_cache(debug=debug)
317                return True, f"{p} is being synced in parallel."
318
319        ### CHECKPOINT: Retrieved the DataFrame.
320        _checkpoint(**kw)
321
322        ### Allow for dataframe generators or iterables.
323        if df_is_chunk_generator(df):
324            kw['workers'] = p.get_num_workers(kw.get('workers', None))
325            dt_col = p.columns.get('datetime', None)
326            pool = get_pool(workers=kw.get('workers', 1))
327            if debug:
328                dprint(f"Received {type(df)}. Attempting to sync first chunk...")
329
330            try:
331                chunk = next(df)
332            except StopIteration:
333                return True, "Received an empty generator; nothing to do."
334
335            chunk_success, chunk_msg = _sync(p, chunk)
336            chunk_msg = '\n' + self._get_chunk_label(chunk, dt_col) + '\n' + chunk_msg
337            if not chunk_success:
338                return chunk_success, f"Unable to sync initial chunk for {p}:\n{chunk_msg}"
339            if debug:
340                dprint("Successfully synced the first chunk, attemping the rest...")
341
342            def _process_chunk(_chunk):
343                _chunk_attempts = 0
344                _max_chunk_attempts = 3
345                while _chunk_attempts < _max_chunk_attempts:
346                    try:
347                        _chunk_success, _chunk_msg = _sync(p, _chunk)
348                    except Exception as e:
349                        _chunk_success, _chunk_msg = False, str(e)
350                    if _chunk_success:
351                        break
352                    _chunk_attempts += 1
353                    _sleep_seconds = _chunk_attempts ** 2
354                    warn(
355                        (
356                            f"Failed to sync chunk to {self} "
357                            + f"(attempt {_chunk_attempts} / {_max_chunk_attempts}).\n"
358                            + f"Sleeping for {_sleep_seconds} second"
359                            + ('s' if _sleep_seconds != 1 else '')
360                            + f":\n{_chunk_msg}"
361                        ),
362                        stack=False,
363                    )
364                    time.sleep(_sleep_seconds)
365
366                num_rows_str = (
367                    f"{num_rows:,} rows"
368                    if (num_rows := len(_chunk)) != 1
369                    else f"{num_rows} row"
370                )
371                _chunk_msg = (
372                    (
373                        "Synced"
374                        if _chunk_success
375                        else "Failed to sync"
376                    ) + f" a chunk ({num_rows_str}) to {p}:\n"
377                    + self._get_chunk_label(_chunk, dt_col)
378                    + '\n'
379                    + _chunk_msg
380                )
381
382                mrsm.pprint((_chunk_success, _chunk_msg), calm=True)
383                return _chunk_success, _chunk_msg
384
385            results = sorted(
386                [(chunk_success, chunk_msg)] + (
387                    list(pool.imap(_process_chunk, df))
388                    if (
389                        not df_is_chunk_generator(chunk)  # Handle nested generators.
390                        and kw.get('workers', 1) != 1
391                    )
392                    else list(
393                        _process_chunk(_child_chunks)
394                        for _child_chunks in df
395                    )
396                )
397            )
398            chunk_messages = [chunk_msg for _, chunk_msg in results]
399            success_bools = [chunk_success for chunk_success, _ in results]
400            num_successes = len([chunk_success for chunk_success, _ in results if chunk_success])
401            num_failures = len([chunk_success for chunk_success, _ in results if not chunk_success])
402            success = all(success_bools)
403            msg = (
404                'Synced '
405                + f'{len(chunk_messages):,} chunk'
406                + ('s' if len(chunk_messages) != 1 else '')
407                + f' to {p}\n({num_successes} succeeded, {num_failures} failed):\n\n'
408                + '\n\n'.join(chunk_messages).lstrip().rstrip()
409            ).lstrip().rstrip()
410            return success, msg
411
412        ### Cast to a dataframe and ensure datatypes are what we expect.
413        dtypes = p.get_dtypes(debug=debug)
414        df = p.enforce_dtypes(
415            df,
416            chunksize=chunksize,
417            enforce=enforce_dtypes,
418            dtypes=dtypes,
419            debug=debug,
420        )
421        if p.autotime:
422            dt_col = p.columns.get('datetime', None)
423            ts_col = dt_col or mrsm.get_config(
424                'pipes', 'autotime', 'column_name_if_datetime_missing'
425            )
426            ts_typ = dtypes.get(ts_col, 'datetime') if ts_col else 'datetime'
427            if ts_col and hasattr(df, 'columns') and ts_col not in df.columns:
428                precision = p.get_precision(debug=debug)
429                now = get_current_timestamp(
430                    precision_unit=precision.get(
431                        'unit',
432                        STATIC_CONFIG['dtypes']['datetime']['default_precision_unit']
433                    ),
434                    precision_interval=precision.get('interval', 1),
435                    round_to=(precision.get('round_to', 'down')),
436                    as_int=(are_dtypes_equal(ts_typ, 'int')),
437                )
438                if debug:
439                    dprint(f"Adding current timestamp to dataframe synced to {p}: {now}")
440
441                df[ts_col] = now
442                kw['check_existing'] = dt_col is not None
443
444        ### Capture special columns.
445        capture_success, capture_msg = self._persist_new_special_columns(
446            df,
447            dtypes=dtypes,
448            debug=debug,
449        )
450        if not capture_success:
451            warn(f"Failed to capture new special columns for {self}:\n{capture_msg}")
452
453        if debug:
454            dprint(
455                "DataFrame to sync:\n"
456                + (
457                    str(df)[:255]
458                    + '...'
459                    if len(str(df)) >= 256
460                    else str(df)
461                ),
462                **kw
463            )
464
465        ### if force, continue to sync until success
466        return_tuple = False, f"Did not sync {p}."
467        run = True
468        _retries = 1
469        while run:
470            with Venv(get_connector_plugin(self.instance_connector)):
471                return_tuple = p.instance_connector.sync_pipe(
472                    pipe=p,
473                    df=df,
474                    debug=debug,
475                    **kw
476                )
477            _retries += 1
478            run = (not return_tuple[0]) and force and _retries <= retries
479            if run and debug:
480                dprint(f"Syncing failed for {p}. Attempt ( {_retries} / {retries} )", **kw)
481                dprint(f"Sleeping for {min_seconds} seconds...", **kw)
482                time.sleep(min_seconds)
483            if _retries > retries:
484                warn(
485                    f"Unable to sync {p} within {retries} attempt" +
486                        ("s" if retries != 1 else "") + "!"
487                )
488
489        ### CHECKPOINT: Finished syncing.
490        _checkpoint(**kw)
491        p._invalidate_cache(debug=debug)
492
493        ### Automatically apply a compression policy if the pipe is configured for compression.
494        if return_tuple[0] and p.parameters.get('compress', False):
495            if hasattr(p.instance_connector, 'apply_compression_policy'):
496                try:
497                    with Venv(get_connector_plugin(p.instance_connector)):
498                        compress_success, compress_msg = (
499                            p.instance_connector.apply_compression_policy(p, debug=debug)
500                        )
501                    if not compress_success and debug:
502                        dprint(f"Could not apply compression policy to {p}:\n{compress_msg}")
503                except Exception as compress_e:
504                    warn(
505                        f"Failed to apply compression policy to {p}:\n{compress_e}",
506                        stack=False,
507                    )
508
509        return return_tuple
510
511    if blocking:
512        return _sync(self, df=df)
513
514    from meerschaum.utils.threading import Thread
515    def default_callback(result_tuple: SuccessTuple):
516        dprint(f"Asynchronous result from {self}: {result_tuple}", **kw)
517
518    def default_error_callback(x: Exception):
519        dprint(f"Error received for {self}: {x}", **kw)
520
521    if callback is None and debug:
522        callback = default_callback
523    if error_callback is None and debug:
524        error_callback = default_error_callback
525    try:
526        thread = Thread(
527            target=_sync,
528            args=(self,),
529            kwargs={'df': df},
530            daemon=False,
531            callback=callback,
532            error_callback=error_callback,
533        )
534        thread.start()
535    except Exception as e:
536        self._invalidate_cache(debug=debug)
537        return False, str(e)
538
539    self._invalidate_cache(debug=debug)
540    return True, f"Spawned asyncronous sync for {self}."

Fetch new data from the source and update the pipe's table with new data.

Get new remote data via fetch, get existing data in the same time period, and merge the two, only keeping the unseen data.

Parameters
  • df (Union[None, pd.DataFrame, Dict[str, List[Any]]], default None): An optional DataFrame to sync into the pipe. Defaults to None. If df is a string, it will be parsed via meerschaum.utils.dataframe.parse_simple_lines().
  • begin (Union[datetime, int, str, None], default ''): Optionally specify the earliest datetime to search for data.
  • end (Union[datetime, int, str, None], default None): Optionally specify the latest datetime to search for data.
  • force (bool, default False): If True, keep trying to sync untul retries attempts.
  • retries (int, default 10): If force, how many attempts to try syncing before declaring failure.
  • min_seconds (Union[int, float], default 1): If force, how many seconds to sleep between retries. Defaults to 1.
  • check_existing (bool, default True): If True, pull and diff with existing data from the pipe.
  • enforce_dtypes (bool, default True): If True, enforce dtypes on incoming data. Set this to False if the incoming rows are expected to be of the correct dtypes.
  • blocking (bool, default True): If True, wait for sync to finish and return its result, otherwise asyncronously sync (oxymoron?) and return success. Defaults to True. Only intended for specific scenarios.
  • workers (Optional[int], default None): If provided and the instance connector is thread-safe (pipe.instance_connector.IS_THREAD_SAFE is True), limit concurrent sync to this many threads.
  • callback (Optional[Callable[[Tuple[bool, str]], Any]], default None): Callback function which expects a SuccessTuple as input. Only applies when blocking=False.
  • error_callback (Optional[Callable[[Exception], Any]], default None): Callback function which expects an Exception as input. Only applies when blocking=False.
  • chunksize (int, default -1): Specify the number of rows to sync per chunk. If -1, resort to system configuration (default is 900). A chunksize of None will sync all rows in one transaction.
  • sync_chunks (bool, default True): If possible, sync chunks while fetching them into memory.
  • debug (bool, default False): Verbosity toggle. Defaults to False.
Returns
def get_sync_time( self, params: Optional[Dict[str, Any]] = None, newest: bool = True, apply_backtrack_interval: bool = False, remote: bool = False, round_down: bool = False, debug: bool = False) -> Union[datetime.datetime, int, NoneType]:
543def get_sync_time(
544    self,
545    params: Optional[Dict[str, Any]] = None,
546    newest: bool = True,
547    apply_backtrack_interval: bool = False,
548    remote: bool = False,
549    round_down: bool = False,
550    debug: bool = False
551) -> Union['datetime', int, None]:
552    """
553    Get the most recent datetime value for a Pipe.
554
555    Parameters
556    ----------
557    params: Optional[Dict[str, Any]], default None
558        Dictionary to build a WHERE clause for a specific column.
559        See `meerschaum.utils.sql.build_where`.
560
561    newest: bool, default True
562        If `True`, get the most recent datetime (honoring `params`).
563        If `False`, get the oldest datetime (`ASC` instead of `DESC`).
564
565    apply_backtrack_interval: bool, default False
566        If `True`, subtract the backtrack interval from the sync time.
567
568    remote: bool, default False
569        If `True` and the instance connector supports it, return the sync time
570        for the remote table definition.
571
572    round_down: bool, default False
573        If `True`, round down the datetime value to the nearest minute.
574
575    debug: bool, default False
576        Verbosity toggle.
577
578    Returns
579    -------
580    A `datetime` or int, if the pipe exists, otherwise `None`.
581
582    """
583    from meerschaum.utils.venv import Venv
584    from meerschaum.connectors import get_connector_plugin
585    from meerschaum.utils.misc import filter_keywords
586    from meerschaum.utils.dtypes import round_time
587    from meerschaum.utils.warnings import warn
588
589    if not self.columns.get('datetime', None):
590        return None
591
592    connector = self.instance_connector if not remote else self.connector
593    if isinstance(connector, str) or connector is None:
594        return None
595
596    with Venv(get_connector_plugin(connector)):
597        if not hasattr(connector, 'get_sync_time'):
598            warn(
599                f"Connectors of type '{connector.type}' "
600                "do not implement `get_sync_time().",
601                stack=False,
602            )
603            return None
604        sync_time = connector.get_sync_time(
605            self,
606            **filter_keywords(
607                connector.get_sync_time,
608                params=params,
609                newest=newest,
610                remote=remote,
611                debug=debug,
612            )
613        )
614
615    if round_down and isinstance(sync_time, datetime):
616        sync_time = round_time(sync_time, timedelta(minutes=1))
617
618    if apply_backtrack_interval and sync_time is not None:
619        backtrack_interval = self.get_backtrack_interval(debug=debug)
620        try:
621            sync_time -= backtrack_interval
622        except Exception as e:
623            warn(f"Failed to apply backtrack interval:\n{e}")
624
625    return self.parse_date_bounds(sync_time)

Get the most recent datetime value for a Pipe.

Parameters
  • params (Optional[Dict[str, Any]], default None): Dictionary to build a WHERE clause for a specific column. See meerschaum.utils.sql.build_where.
  • newest (bool, default True): If True, get the most recent datetime (honoring params). If False, get the oldest datetime (ASC instead of DESC).
  • apply_backtrack_interval (bool, default False): If True, subtract the backtrack interval from the sync time.
  • remote (bool, default False): If True and the instance connector supports it, return the sync time for the remote table definition.
  • round_down (bool, default False): If True, round down the datetime value to the nearest minute.
  • debug (bool, default False): Verbosity toggle.
Returns
  • A datetime or int, if the pipe exists, otherwise None.
def exists(self, debug: bool = False) -> bool:
628def exists(
629    self,
630    debug: bool = False
631) -> bool:
632    """
633    See if a Pipe's table exists.
634
635    Parameters
636    ----------
637    debug: bool, default False
638        Verbosity toggle.
639
640    Returns
641    -------
642    A `bool` corresponding to whether a pipe's underlying table exists.
643
644    """
645    from meerschaum.utils.venv import Venv
646    from meerschaum.connectors import get_connector_plugin
647    from meerschaum.utils.debug import dprint
648    from meerschaum.utils.dtypes import get_current_timestamp
649    now = get_current_timestamp('ms', as_int=True) / 1000
650    cache_seconds = mrsm.get_config('pipes', 'sync', 'exists_cache_seconds')
651
652    _exists = self._get_cached_value('_exists', debug=debug)
653    if _exists:
654        exists_timestamp = self._get_cached_value('_exists_timestamp', debug=debug)
655        if exists_timestamp is not None:
656            delta = now - exists_timestamp
657            if delta < cache_seconds:
658                if debug:
659                    dprint(f"Returning cached `exists` for {self} ({round(delta, 2)} seconds old).")
660                return _exists
661
662    with Venv(get_connector_plugin(self.instance_connector)):
663        _exists = (
664            self.instance_connector.pipe_exists(pipe=self, debug=debug)
665            if hasattr(self.instance_connector, 'pipe_exists')
666            else False
667        )
668
669    self._cache_value('_exists', _exists, debug=debug)
670    self._cache_value('_exists_timestamp', now, debug=debug)
671    return _exists

See if a Pipe's table exists.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
  • A bool corresponding to whether a pipe's underlying table exists.
def filter_existing( self, df: pandas.DataFrame, safe_copy: bool = True, date_bound_only: bool = False, include_unchanged_columns: bool = False, enforce_dtypes: bool = False, chunksize: Optional[int] = -1, debug: bool = False, **kw) -> Tuple[pandas.DataFrame, pandas.DataFrame, pandas.DataFrame]:
674def filter_existing(
675    self,
676    df: 'pd.DataFrame',
677    safe_copy: bool = True,
678    date_bound_only: bool = False,
679    include_unchanged_columns: bool = False,
680    enforce_dtypes: bool = False,
681    chunksize: Optional[int] = -1,
682    debug: bool = False,
683    **kw
684) -> Tuple['pd.DataFrame', 'pd.DataFrame', 'pd.DataFrame']:
685    """
686    Inspect a dataframe and filter out rows which already exist in the pipe.
687
688    Parameters
689    ----------
690    df: 'pd.DataFrame'
691        The dataframe to inspect and filter.
692        A `polars.DataFrame` or `LazyFrame` is accepted,
693        in which case the returned dataframes are `polars.DataFrame`.
694
695    safe_copy: bool, default True
696        If `True`, create a copy before comparing and modifying the dataframes.
697        Setting to `False` may mutate the DataFrames.
698        See `meerschaum.utils.dataframe.filter_unseen_df`.
699
700    date_bound_only: bool, default False
701        If `True`, only use the datetime index to fetch the sample dataframe.
702
703    include_unchanged_columns: bool, default False
704        If `True`, include the backtrack columns which haven't changed in the update dataframe.
705        This is useful if you can't update individual keys.
706
707    enforce_dtypes: bool, default False
708        If `True`, ensure the given and intermediate dataframes are enforced to the correct dtypes.
709        Setting `enforce_dtypes=True` may impact performance.
710
711    chunksize: Optional[int], default -1
712        The `chunksize` used when fetching existing data.
713
714    debug: bool, default False
715        Verbosity toggle.
716
717    Returns
718    -------
719    A tuple of three DataFrames: unseen, update, and delta.
720    These are Polars DataFrames if `df` is a Polars DataFrame or LazyFrame, else Pandas.
721    """
722    from meerschaum.utils.warnings import warn
723    from meerschaum.utils.debug import dprint
724    from meerschaum.utils.packages import attempt_import, import_pandas
725    from meerschaum.utils.dataframe import (
726        filter_unseen_df,
727        add_missing_cols_to_df,
728        get_unhashable_cols,
729        get_json_cols,
730        to_pandas,
731        to_polars,
732    )
733    from meerschaum.utils.dtypes import (
734        to_pandas_dtype,
735        none_if_null,
736    )
737    from meerschaum.config import get_config
738    pd = import_pandas()
739    pandas = attempt_import('pandas')
740    ### Polars frames are filtered natively when possible;
741    ### otherwise they are converted up-front and converted back on the way out.
742    input_is_polars = df is not None and df.__class__.__module__.split('.')[0] == 'polars'
743    if input_is_polars:
744        polars_dfs = self._filter_existing_polars(
745            df,
746            safe_copy=safe_copy,
747            date_bound_only=date_bound_only,
748            include_unchanged_columns=include_unchanged_columns,
749            enforce_dtypes=enforce_dtypes,
750            chunksize=chunksize,
751            debug=debug,
752            **kw
753        )
754        if polars_dfs is not None:
755            return polars_dfs
756
757    df = to_pandas(df)
758
759    def _return_dfs(unseen, update, delta):
760        if not input_is_polars:
761            return unseen, update, delta
762        return tuple(
763            to_polars(_df, json_cols=get_json_cols(_df))
764            for _df in (unseen, update, delta)
765        )
766
767    if enforce_dtypes or 'dataframe' not in str(type(df)).lower():
768        df = self.enforce_dtypes(df, chunksize=chunksize, debug=debug)
769    is_dask = hasattr(df, '__module__') and 'dask' in df.__module__
770    if is_dask:
771        dd = attempt_import('dask.dataframe')
772        merge = dd.merge
773        NA = pandas.NA
774    else:
775        merge = pd.merge
776        NA = pd.NA
777
778    parameters = self.parameters
779    pipe_columns = self.columns
780    primary_key = pipe_columns.get('primary', None)
781    dt_col = pipe_columns.get('datetime', None)
782    dt_type = parameters.get('dtypes', {}).get(dt_col, 'datetime') if dt_col else None
783    autoincrement = parameters.get('autoincrement', False)
784    autotime = parameters.get('autotime', False)
785
786    if primary_key and autoincrement and df is not None and primary_key in df.columns:
787        if safe_copy:
788            df = df.copy()
789            safe_copy = False
790        if df[primary_key].isnull().all():
791            del df[primary_key]
792            _ = self.columns.pop(primary_key, None)
793
794    if dt_col and autotime and df is not None and dt_col in df.columns:
795        if safe_copy:
796            df = df.copy()
797            safe_copy = False
798        if df[dt_col].isnull().all():
799            del df[dt_col]
800            _ = self.columns.pop(dt_col, None)
801
802    def get_empty_df():
803        empty_df = pd.DataFrame([])
804        dtypes = dict(df.dtypes) if df is not None else {}
805        dtypes.update(self.dtypes) if self.enforce else {}
806        pd_dtypes = {
807            col: to_pandas_dtype(str(typ))
808            for col, typ in dtypes.items()
809        }
810        return add_missing_cols_to_df(empty_df, pd_dtypes)
811
812    if df is None:
813        empty_df = get_empty_df()
814        return _return_dfs(empty_df, empty_df, empty_df)
815
816    if (df.empty if not is_dask else len(df) == 0):
817        return _return_dfs(df, df, df)
818
819    if autoincrement and primary_key == dt_col and dt_col not in df.columns:
820        if enforce_dtypes:
821            df = self.enforce_dtypes(df, chunksize=chunksize, debug=debug)
822        return _return_dfs(df, get_empty_df(), df)
823
824    if autotime and dt_col and dt_col not in df.columns:
825        if enforce_dtypes:
826            df = self.enforce_dtypes(df, chunksize=chunksize, debug=debug)
827        return _return_dfs(df, get_empty_df(), df)
828
829    try:
830        min_dt_val = df[dt_col].min(skipna=True) if dt_col and dt_col in df.columns else None
831        max_dt_val = df[dt_col].max(skipna=True) if dt_col and dt_col in df.columns else None
832        if is_dask:
833            min_dt_val = min_dt_val.compute() if min_dt_val is not None else None
834            max_dt_val = max_dt_val.compute() if max_dt_val is not None else None
835    except Exception:
836        min_dt_val, max_dt_val = None, None
837
838    ### `begin` is the oldest data in the new dataframe, `end` is the newest.
839    begin, end = _get_delta_bounds(min_dt_val, max_dt_val, dt_type, dt_col)
840
841    unique_index_vals = {
842        col: df[col].unique()
843        for col in (pipe_columns.values() if not primary_key else [primary_key])
844        if col in df.columns and col != dt_col
845    } if not date_bound_only else {}
846    unique_index_lens = {
847        col: len(unique_vals)
848        for col, unique_vals in unique_index_vals.items()
849    } if not date_bound_only else {}
850    filter_params_index_limit = get_config('pipes', 'sync', 'filter_params_index_limit')
851    _ = kw.pop('params', None)
852    params = {
853        col: [
854            none_if_null(val)
855            for val in unique_vals
856        ]
857        for col, unique_vals in unique_index_vals.items()
858        if unique_index_lens[col] <= filter_params_index_limit
859    } if not date_bound_only else {}
860
861    if debug:
862        dprint(
863            (
864                f"Looking at data between '{begin}' and '{end}' with index value lengths:\n"
865                f"{json.dumps(unique_index_lens, indent=4)}\n"
866            ),
867            **kw
868        )
869
870    backtrack_df = self.get_data(
871        begin=begin,
872        end=end,
873        chunksize=chunksize,
874        params=params,
875        debug=debug,
876        **kw
877    )
878    if backtrack_df is None:
879        if debug:
880            dprint(f"No backtrack data was found for {self}.")
881        return _return_dfs(df, get_empty_df(), df)
882
883    if enforce_dtypes:
884        backtrack_df = self.enforce_dtypes(backtrack_df, chunksize=chunksize, debug=debug)
885
886    if debug:
887        dprint(f"Existing data for {self}:\n" + str(backtrack_df), **kw)
888        dprint(f"Existing dtypes for {self}:\n" + str(backtrack_df.dtypes))
889
890    ### Separate new rows from changed ones.
891    on_cols = [
892        col
893        for col_key, col in pipe_columns.items()
894        if (
895            col
896            and
897            col_key != 'value'
898            and col in backtrack_df.columns
899        )
900    ] if not primary_key else [primary_key]
901
902    self_dtypes = self.get_dtypes(debug=debug) if self.enforce else {}
903    on_cols_dtypes = {
904        col: to_pandas_dtype(typ)
905        for col, typ in self_dtypes.items()
906        if col in on_cols
907    }
908
909    ### Detect changes between the old target and new source dataframes.
910    delta_df = add_missing_cols_to_df(
911        filter_unseen_df(
912            backtrack_df,
913            df,
914            dtypes={
915                col: to_pandas_dtype(typ)
916                for col, typ in self_dtypes.items()
917            },
918            safe_copy=safe_copy,
919            coerce_mixed_numerics=(not self.static),
920            debug=debug
921        ),
922        on_cols_dtypes,
923    )
924    if enforce_dtypes:
925        delta_df = self.enforce_dtypes(delta_df, chunksize=chunksize, debug=debug)
926
927    for col, typ in on_cols_dtypes.items():
928        delta_df[col] = delta_df[col].astype(typ)
929        backtrack_df[col] = backtrack_df[col].astype(typ)
930
931    ### Cast dicts or lists to strings so we can merge.
932    serializer = functools.partial(json.dumps, sort_keys=True, separators=(',', ':'), default=str)
933
934    def deserializer(x):
935        return json.loads(x) if isinstance(x, str) else x
936
937    unhashable_delta_cols = get_unhashable_cols(delta_df)
938    unhashable_backtrack_cols = get_unhashable_cols(backtrack_df)
939    for col in unhashable_delta_cols:
940        delta_df[col] = delta_df[col].apply(serializer)
941    for col in unhashable_backtrack_cols:
942        backtrack_df[col] = backtrack_df[col].apply(serializer)
943    casted_cols = set(unhashable_delta_cols + unhashable_backtrack_cols)
944
945    joined_df = merge(
946        delta_df.infer_objects().fillna(NA),
947        backtrack_df.infer_objects().fillna(NA),
948        how='left',
949        on=on_cols,
950        indicator=True,
951        suffixes=('', '_old'),
952    ) if on_cols else delta_df
953    for col in casted_cols:
954        if col in joined_df.columns:
955            joined_df[col] = joined_df[col].apply(deserializer)
956        if col in delta_df.columns:
957            delta_df[col] = delta_df[col].apply(deserializer)
958
959    ### Determine which rows are completely new.
960    new_rows_mask = (joined_df['_merge'] == 'left_only') if on_cols else None
961    cols = list(delta_df.columns)
962
963    unseen_df = (
964        joined_df[new_rows_mask][cols].reset_index(drop=True)
965    ) if on_cols else delta_df
966
967    ### Rows that have already been inserted but values have changed.
968    update_df = (
969        joined_df[~new_rows_mask][cols].reset_index(drop=True)
970    ) if on_cols else get_empty_df()
971
972    if include_unchanged_columns and on_cols:
973        unchanged_backtrack_cols = [
974            col
975            for col in backtrack_df.columns
976            if col in on_cols or col not in update_df.columns
977        ]
978        if enforce_dtypes:
979            update_df = self.enforce_dtypes(update_df, chunksize=chunksize, debug=debug)
980        update_df = merge(
981            backtrack_df[unchanged_backtrack_cols],
982            update_df,
983            how='inner',
984            on=on_cols,
985        )
986
987    return _return_dfs(unseen_df, update_df, delta_df)

Inspect a dataframe and filter out rows which already exist in the pipe.

Parameters
  • df ('pd.DataFrame'): The dataframe to inspect and filter. A polars.DataFrame or LazyFrame is accepted, in which case the returned dataframes are polars.DataFrame.
  • safe_copy (bool, default True): If True, create a copy before comparing and modifying the dataframes. Setting to False may mutate the DataFrames. See meerschaum.utils.dataframe.filter_unseen_df.
  • date_bound_only (bool, default False): If True, only use the datetime index to fetch the sample dataframe.
  • include_unchanged_columns (bool, default False): If True, include the backtrack columns which haven't changed in the update dataframe. This is useful if you can't update individual keys.
  • enforce_dtypes (bool, default False): If True, ensure the given and intermediate dataframes are enforced to the correct dtypes. Setting enforce_dtypes=True may impact performance.
  • chunksize (Optional[int], default -1): The chunksize used when fetching existing data.
  • debug (bool, default False): Verbosity toggle.
Returns
  • A tuple of three DataFrames (unseen, update, and delta.):

  • These are Polars DataFrames if df is a Polars DataFrame or LazyFrame, else Pandas.

def get_num_workers(self, workers: Optional[int] = None) -> int:
1381def get_num_workers(self, workers: Optional[int] = None) -> int:
1382    """
1383    Get the number of workers to use for concurrent syncs.
1384
1385    Parameters
1386    ----------
1387    The number of workers passed via `--workers`.
1388
1389    Returns
1390    -------
1391    The number of workers, capped for safety.
1392    """
1393    is_thread_safe = getattr(self.instance_connector, 'IS_THREAD_SAFE', False)
1394    if not is_thread_safe:
1395        return 1
1396
1397    engine_pool_size = (
1398        self.instance_connector.engine.pool.size()
1399        if self.instance_connector.type == 'sql'
1400        else None
1401    )
1402    current_num_threads = threading.active_count()
1403    current_num_connections = (
1404        self.instance_connector.engine.pool.checkedout()
1405        if engine_pool_size is not None
1406        else current_num_threads
1407    )
1408    desired_workers = (
1409        min(workers or engine_pool_size, engine_pool_size)
1410        if engine_pool_size is not None
1411        else workers
1412    )
1413    if desired_workers is None:
1414        desired_workers = (multiprocessing.cpu_count() if is_thread_safe else 1)
1415
1416    return max(
1417        (desired_workers - current_num_connections),
1418        1,
1419    )

Get the number of workers to use for concurrent syncs.

Parameters
  • The number of workers passed via --workers.
Returns
  • The number of workers, capped for safety.
def verify( self, begin: Union[datetime.datetime, int, NoneType] = None, end: Union[datetime.datetime, int, NoneType] = None, params: Optional[Dict[str, Any]] = None, chunk_interval: Union[datetime.timedelta, int, NoneType] = None, bounded: Optional[bool] = None, deduplicate: bool = False, workers: Optional[int] = None, batchsize: Optional[int] = None, skip_chunks_with_greater_rowcounts: bool = False, check_rowcounts_only: bool = False, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
 19def verify(
 20    self,
 21    begin: Union[datetime, int, None] = None,
 22    end: Union[datetime, int, None] = None,
 23    params: Optional[Dict[str, Any]] = None,
 24    chunk_interval: Union[timedelta, int, None] = None,
 25    bounded: Optional[bool] = None,
 26    deduplicate: bool = False,
 27    workers: Optional[int] = None,
 28    batchsize: Optional[int] = None,
 29    skip_chunks_with_greater_rowcounts: bool = False,
 30    check_rowcounts_only: bool = False,
 31    debug: bool = False,
 32    **kwargs: Any
 33) -> SuccessTuple:
 34    """
 35    Verify the contents of the pipe by resyncing its interval.
 36
 37    Parameters
 38    ----------
 39    begin: Union[datetime, int, None], default None
 40        If specified, only verify rows greater than or equal to this value.
 41
 42    end: Union[datetime, int, None], default None
 43        If specified, only verify rows less than this value.
 44
 45    chunk_interval: Union[timedelta, int, None], default None
 46        If provided, use this as the size of the chunk boundaries.
 47        Default to the value set in `pipe.parameters['verify']['chunk_minutes']` (43200 — 30 days).
 48
 49    bounded: Optional[bool], default None
 50        If `True`, do not verify older than the oldest sync time or newer than the newest.
 51        If `False`, verify unbounded syncs outside of the new and old sync times.
 52        The default behavior (`None`) is to bound only if a bound interval is set
 53        (e.g. `pipe.parameters['verify']['bound_days']`).
 54
 55    deduplicate: bool, default False
 56        If `True`, deduplicate the pipe's table after the verification syncs.
 57
 58    workers: Optional[int], default None
 59        If provided, limit the verification to this many threads.
 60        Use a value of `1` to sync chunks in series.
 61
 62    batchsize: Optional[int], default None
 63        If provided, sync this many chunks in parallel.
 64        Defaults to `Pipe.get_num_workers()`.
 65
 66    skip_chunks_with_greater_rowcounts: bool, default False
 67        If `True`, compare the rowcounts for a chunk and skip syncing if the pipe's
 68        chunk rowcount equals or exceeds the remote's rowcount.
 69
 70    check_rowcounts_only: bool, default False
 71        If `True`, only compare rowcounts and print chunks which are out-of-sync.
 72
 73    debug: bool, default False
 74        Verbosity toggle.
 75
 76    kwargs: Any
 77        All keyword arguments are passed to `pipe.sync()`.
 78
 79    Returns
 80    -------
 81    A SuccessTuple indicating whether the pipe was successfully resynced.
 82    """
 83    from meerschaum.utils.pool import get_pool
 84    from meerschaum.utils.formatting import make_header
 85    from meerschaum.utils.misc import interval_str
 86    workers = self.get_num_workers(workers)
 87    check_rowcounts = skip_chunks_with_greater_rowcounts or check_rowcounts_only
 88
 89    ### Skip configured bounding in parameters
 90    ### if `bounded` is explicitly `False`.
 91    bound_time = (
 92        self.get_bound_time(debug=debug)
 93        if bounded is not False
 94        else None
 95    )
 96    if bounded is None:
 97        bounded = bound_time is not None
 98
 99    if bounded and begin is None:
100        begin = (
101            bound_time
102            if bound_time is not None
103            else self.get_sync_time(newest=False, debug=debug)
104        )
105        if begin is None:
106            remote_oldest_sync_time = self.get_sync_time(newest=False, remote=True, debug=debug)
107            begin = remote_oldest_sync_time
108    if bounded and end is None:
109        end = self.get_sync_time(newest=True, debug=debug)
110        if end is None:
111            remote_newest_sync_time = self.get_sync_time(newest=True, remote=True, debug=debug)
112            end = remote_newest_sync_time
113        if end is not None:
114            end += (
115                timedelta(minutes=1)
116                if hasattr(end, 'tzinfo')
117                else 1
118            )
119
120    begin, end = self.parse_date_bounds(begin, end)
121    cannot_determine_bounds = bounded and begin is None and end is None
122
123    if cannot_determine_bounds and not check_rowcounts_only:
124        warn(f"Cannot determine sync bounds for {self}. Syncing instead...", stack=False)
125        sync_success, sync_msg = self.sync(
126            begin=begin,
127            end=end,
128            params=params,
129            workers=workers,
130            debug=debug,
131            **kwargs
132        )
133        if not sync_success:
134            return sync_success, sync_msg
135
136        if deduplicate:
137            return self.deduplicate(
138                begin=begin,
139                end=end,
140                params=params,
141                workers=workers,
142                debug=debug,
143                **kwargs
144            )
145        return sync_success, sync_msg
146
147    chunk_interval = self.get_chunk_interval(chunk_interval, debug=debug)
148    chunk_bounds = self.get_chunk_bounds(
149        begin=begin,
150        end=end,
151        chunk_interval=chunk_interval,
152        bounded=bounded,
153        align=True,
154        debug=debug,
155    )
156
157    ### Consider it a success if no chunks need to be verified.
158    if not chunk_bounds:
159        if deduplicate:
160            return self.deduplicate(
161                begin=begin,
162                end=end,
163                params=params,
164                workers=workers,
165                debug=debug,
166                **kwargs
167            )
168        return True, f"Could not determine chunks between '{begin}' and '{end}'; nothing to do."
169
170    begin_to_print = (
171        begin
172        if begin is not None
173        else (
174            chunk_bounds[0][0]
175            if bounded
176            else chunk_bounds[0][1]
177        )
178    )
179    end_to_print = (
180        end
181        if end is not None
182        else (
183            chunk_bounds[-1][1]
184            if bounded
185            else chunk_bounds[-1][0]
186        )
187    )
188    message_header = f"{begin_to_print} - {end_to_print}"
189    max_chunks_syncs = mrsm.get_config('pipes', 'verify', 'max_chunks_syncs')
190
191    info(
192        f"Verifying {self}:\n    "
193        + ("Syncing" if not check_rowcounts_only else "Checking")
194        + f" {len(chunk_bounds)} chunk"
195        + ('s' if len(chunk_bounds) != 1 else '')
196        + f" ({'un' if not bounded else ''}bounded)"
197        + f" of size '{interval_str(chunk_interval)}'"
198        + f" between '{begin_to_print}' and '{end_to_print}'.\n"
199    )
200
201    ### Dictionary of the form bounds -> success_tuple, e.g.:
202    ### {
203    ###    (2023-01-01, 2023-01-02): (True, "Success")
204    ### }
205    bounds_success_tuples = {}
206    def process_chunk_bounds(
207        chunk_begin_and_end: Tuple[
208            Union[int, datetime],
209            Union[int, datetime]
210        ],
211        _workers: Optional[int] = 1,
212    ):
213        if chunk_begin_and_end in bounds_success_tuples:
214            return chunk_begin_and_end, bounds_success_tuples[chunk_begin_and_end]
215
216        chunk_begin, chunk_end = chunk_begin_and_end
217        do_sync = True
218        chunk_success, chunk_msg = False, "Did not sync chunk."
219        if check_rowcounts:
220            existing_rowcount = self.get_rowcount(begin=chunk_begin, end=chunk_end, debug=debug)
221            remote_rowcount = self.get_rowcount(
222                begin=chunk_begin,
223                end=chunk_end,
224                remote=True,
225                debug=debug,
226            )
227            checked_rows_str = (
228                f"checked {existing_rowcount:,} row"
229                + ("s" if existing_rowcount != 1 else '')
230                + f" vs {remote_rowcount:,} remote"
231            )
232            if (
233                existing_rowcount is not None
234                and remote_rowcount is not None
235                and existing_rowcount >= remote_rowcount
236            ):
237                do_sync = False
238                chunk_success, chunk_msg = True, (
239                    "Row-count is up-to-date "
240                    f"({checked_rows_str})."
241                )
242            elif check_rowcounts_only:
243                do_sync = False
244                chunk_success, chunk_msg = True, (
245                    f"Row-counts are out-of-sync ({checked_rows_str})."
246                )
247
248        num_syncs = 0
249        while num_syncs < max_chunks_syncs:
250            chunk_success, chunk_msg = self.sync(
251                begin=chunk_begin,
252                end=chunk_end,
253                params=params,
254                workers=_workers,
255                debug=debug,
256                **kwargs
257            ) if do_sync else (chunk_success, chunk_msg)
258            if chunk_success:
259                break
260            num_syncs += 1
261            time.sleep(num_syncs**2)
262        chunk_msg = chunk_msg.strip()
263        if ' - ' not in chunk_msg:
264            chunk_label = f"{chunk_begin} - {chunk_end}"
265            chunk_msg = f'Verified chunk for {self}:\n{chunk_label}\n{chunk_msg}'
266        mrsm.pprint((chunk_success, chunk_msg))
267
268        return chunk_begin_and_end, (chunk_success, chunk_msg)
269
270    ### If we have more than one chunk, attempt to sync the first one and return if its fails.
271    if len(chunk_bounds) > 1:
272        first_chunk_bounds = chunk_bounds[0]
273        first_label = f"{first_chunk_bounds[0]} - {first_chunk_bounds[1]}"
274        info(f"Verifying first chunk for {self}:\n    {first_label}")
275        (
276            (first_begin, first_end),
277            (first_success, first_msg)
278        ) = process_chunk_bounds(first_chunk_bounds, _workers=workers)
279        if not first_success:
280            return (
281                first_success,
282                f"\n{first_label}\n"
283                + f"Failed to sync first chunk:\n{first_msg}"
284            )
285        bounds_success_tuples[first_chunk_bounds] = (first_success, first_msg)
286        info(f"Completed first chunk for {self}:\n    {first_label}\n")
287        chunk_bounds = chunk_bounds[1:]
288
289    pool = get_pool(workers=workers)
290    batches = self.get_chunk_bounds_batches(chunk_bounds, batchsize=batchsize, workers=workers)
291
292    def process_batch(
293        batch_chunk_bounds: Tuple[
294            Tuple[Union[datetime, int, None], Union[datetime, int, None]],
295            ...
296        ]
297    ):
298        _batch_begin = batch_chunk_bounds[0][0]
299        _batch_end = batch_chunk_bounds[-1][-1]
300        batch_message_header = f"{_batch_begin} - {_batch_end}"
301
302        if check_rowcounts_only:
303            info(f"Checking row-counts for batch bounds:\n    {batch_message_header}")
304            _, (batch_init_success, batch_init_msg) = process_chunk_bounds(
305                (_batch_begin, _batch_end)
306            )
307            mrsm.pprint((batch_init_success, batch_init_msg))
308            if batch_init_success and 'up-to-date' in batch_init_msg:
309                info("Entire batch is up-to-date.")
310                return batch_init_success, batch_init_msg
311
312        batch_bounds_success_tuples = dict(pool.map(process_chunk_bounds, batch_chunk_bounds))
313        bounds_success_tuples.update(batch_bounds_success_tuples)
314        batch_bounds_success_bools = {
315            bounds: tup[0]
316            for bounds, tup in batch_bounds_success_tuples.items()
317        }
318
319        if all(batch_bounds_success_bools.values()):
320            msg = get_chunks_success_message(
321                batch_bounds_success_tuples,
322                header=batch_message_header,
323                check_rowcounts_only=check_rowcounts_only,
324            )
325            if deduplicate:
326                deduplicate_success, deduplicate_msg = self.deduplicate(
327                    begin=_batch_begin,
328                    end=_batch_end,
329                    params=params,
330                    workers=workers,
331                    debug=debug,
332                    **kwargs
333                )
334                return deduplicate_success, msg + '\n\n' + deduplicate_msg
335            return True, msg
336
337        batch_chunk_bounds_to_resync = [
338            bounds
339            for bounds, success in zip(batch_chunk_bounds, batch_bounds_success_bools)
340            if not success
341        ]
342        batch_bounds_to_print = [
343            f"{bounds[0]} - {bounds[1]}"
344            for bounds in batch_chunk_bounds_to_resync
345        ]
346        if batch_bounds_to_print:
347            warn(
348                "Will resync the following failed chunks:\n    "
349                + '\n    '.join(batch_bounds_to_print),
350                stack=False,
351            )
352
353        retry_bounds_success_tuples = dict(pool.map(
354            process_chunk_bounds,
355            batch_chunk_bounds_to_resync
356        ))
357        batch_bounds_success_tuples.update(retry_bounds_success_tuples)
358        bounds_success_tuples.update(retry_bounds_success_tuples)
359        retry_bounds_success_bools = {
360            bounds: tup[0]
361            for bounds, tup in retry_bounds_success_tuples.items()
362        }
363
364        if all(retry_bounds_success_bools.values()):
365            chunks_message = (
366                get_chunks_success_message(
367                    batch_bounds_success_tuples,
368                    header=batch_message_header,
369                    check_rowcounts_only=check_rowcounts_only,
370                ) + f"\nRetried {len(batch_chunk_bounds_to_resync)} chunk" + (
371                    's'
372                    if len(batch_chunk_bounds_to_resync) != 1
373                    else ''
374                ) + "."
375            )
376            if deduplicate:
377                deduplicate_success, deduplicate_msg = self.deduplicate(
378                    begin=_batch_begin,
379                    end=_batch_end,
380                    params=params,
381                    workers=workers,
382                    debug=debug,
383                    **kwargs
384                )
385                return deduplicate_success, chunks_message + '\n\n' + deduplicate_msg
386            return True, chunks_message
387
388        batch_chunks_message = get_chunks_success_message(
389            batch_bounds_success_tuples,
390            header=batch_message_header,
391            check_rowcounts_only=check_rowcounts_only,
392        )
393        if deduplicate:
394            deduplicate_success, deduplicate_msg = self.deduplicate(
395                begin=begin,
396                end=end,
397                params=params,
398                workers=workers,
399                debug=debug,
400                **kwargs
401            )
402            return deduplicate_success, batch_chunks_message + '\n\n' + deduplicate_msg
403        return False, batch_chunks_message
404
405    num_batches = len(batches)
406    for batch_i, batch in enumerate(batches):
407        batch_begin = batch[0][0]
408        batch_end = batch[-1][-1]
409        batch_counter_str = f"({(batch_i + 1):,}/{num_batches:,})"
410        batch_label = f"batch {batch_counter_str}:\n{batch_begin} - {batch_end}"
411        retry_failed_batch = True
412        try:
413            for_self = 'for ' + str(self)
414            batch_label_str = batch_label.replace(':\n', ' ' + for_self + '...\n    ')
415            info(f"Verifying {batch_label_str}\n")
416            batch_success, batch_msg = process_batch(batch)
417        except (KeyboardInterrupt, Exception) as e:
418            batch_success = False
419            batch_msg = str(e)
420            retry_failed_batch = False
421
422        batch_msg_to_print = (
423            f"{make_header('Completed batch ' + batch_counter_str + ':', left_pad=0)}\n{batch_msg}"
424        )
425        mrsm.pprint((batch_success, batch_msg_to_print))
426
427        if not batch_success and retry_failed_batch:
428            info(f"Retrying batch {batch_counter_str}...")
429            retry_batch_success, retry_batch_msg = process_batch(batch)
430            retry_batch_msg_to_print = (
431                f"Retried {make_header('batch ' + batch_label, left_pad=0)}\n{retry_batch_msg}"
432            )
433            mrsm.pprint((retry_batch_success, retry_batch_msg_to_print))
434
435            batch_success = retry_batch_success
436            batch_msg = retry_batch_msg
437
438        if not batch_success:
439            return False, f"Failed to verify {batch_label}:\n\n{batch_msg}"
440
441    chunks_message = get_chunks_success_message(
442        bounds_success_tuples,
443        header=message_header,
444        check_rowcounts_only=check_rowcounts_only,
445    )
446    return True, chunks_message

Verify the contents of the pipe by resyncing its interval.

Parameters
  • begin (Union[datetime, int, None], default None): If specified, only verify rows greater than or equal to this value.
  • end (Union[datetime, int, None], default None): If specified, only verify rows less than this value.
  • chunk_interval (Union[timedelta, int, None], default None): If provided, use this as the size of the chunk boundaries. Default to the value set in pipe.parameters['verify']['chunk_minutes'] (43200 — 30 days).
  • bounded (Optional[bool], default None): If True, do not verify older than the oldest sync time or newer than the newest. If False, verify unbounded syncs outside of the new and old sync times. The default behavior (None) is to bound only if a bound interval is set (e.g. pipe.parameters['verify']['bound_days']).
  • deduplicate (bool, default False): If True, deduplicate the pipe's table after the verification syncs.
  • workers (Optional[int], default None): If provided, limit the verification to this many threads. Use a value of 1 to sync chunks in series.
  • batchsize (Optional[int], default None): If provided, sync this many chunks in parallel. Defaults to Pipe.get_num_workers().
  • skip_chunks_with_greater_rowcounts (bool, default False): If True, compare the rowcounts for a chunk and skip syncing if the pipe's chunk rowcount equals or exceeds the remote's rowcount.
  • check_rowcounts_only (bool, default False): If True, only compare rowcounts and print chunks which are out-of-sync.
  • debug (bool, default False): Verbosity toggle.
  • kwargs (Any): All keyword arguments are passed to pipe.sync().
Returns
  • A SuccessTuple indicating whether the pipe was successfully resynced.
def get_bound_interval(self, debug: bool = False) -> Union[datetime.timedelta, int, NoneType]:
547def get_bound_interval(self, debug: bool = False) -> Union[timedelta, int, None]:
548    """
549    Return the interval used to determine the bound time (limit for verification syncs).
550    If the datetime axis is an integer, just return its value.
551
552    Below are the supported keys for the bound interval:
553
554        - `pipe.parameters['verify']['bound_minutes']`
555        - `pipe.parameters['verify']['bound_hours']`
556        - `pipe.parameters['verify']['bound_days']`
557        - `pipe.parameters['verify']['bound_weeks']`
558        - `pipe.parameters['verify']['bound_years']`
559        - `pipe.parameters['verify']['bound_seconds']`
560
561    If multiple keys are present, the first on this priority list will be used.
562
563    Returns
564    -------
565    A `timedelta` or `int` value to be used to determine the bound time.
566    """
567    verify_params = self.parameters.get('verify', {})
568    prefix = 'bound_'
569    suffixes_to_check = ('minutes', 'hours', 'days', 'weeks', 'years', 'seconds')
570    keys_to_search = {
571        key: val
572        for key, val in verify_params.items()
573        if key.startswith(prefix)
574    }
575    bound_time_key, bound_time_value = None, None
576    for key, value in keys_to_search.items():
577        for suffix in suffixes_to_check:
578            if key == prefix + suffix:
579                bound_time_key = key
580                bound_time_value = value
581                break
582        if bound_time_key is not None:
583            break
584
585    if bound_time_value is None:
586        return bound_time_value
587
588    dt_col = self.columns.get('datetime', None)
589    if not dt_col:
590        return bound_time_value
591
592    dt_typ = self.dtypes.get(dt_col, 'datetime')
593    if 'int' in dt_typ.lower():
594        return int(bound_time_value)
595
596    interval_type = bound_time_key.replace(prefix, '')
597    return timedelta(**{interval_type: bound_time_value})

Return the interval used to determine the bound time (limit for verification syncs). If the datetime axis is an integer, just return its value.

Below are the supported keys for the bound interval:

- `pipe.parameters['verify']['bound_minutes']`
- `pipe.parameters['verify']['bound_hours']`
- `pipe.parameters['verify']['bound_days']`
- `pipe.parameters['verify']['bound_weeks']`
- `pipe.parameters['verify']['bound_years']`
- `pipe.parameters['verify']['bound_seconds']`

If multiple keys are present, the first on this priority list will be used.

Returns
  • A timedelta or int value to be used to determine the bound time.
def get_bound_time(self, debug: bool = False) -> Union[datetime.datetime, int, NoneType]:
600def get_bound_time(self, debug: bool = False) -> Union[datetime, int, None]:
601    """
602    The bound time is the limit at which long-running verification syncs should stop.
603    A value of `None` means verification syncs should be unbounded.
604
605    Like deriving a backtrack time from `pipe.get_sync_time()`,
606    the bound time is the sync time minus a large window (e.g. 366 days).
607
608    Unbound verification syncs (i.e. `bound_time is None`)
609    if the oldest sync time is less than the bound interval.
610
611    Returns
612    -------
613    A `datetime` or `int` corresponding to the
614    `begin` bound for verification and deduplication syncs.
615    """
616    bound_interval = self.get_bound_interval(debug=debug)
617    if bound_interval is None:
618        return None
619
620    sync_time = self.get_sync_time(debug=debug)
621    if sync_time is None:
622        return None
623
624    bound_time = sync_time - bound_interval
625    oldest_sync_time = self.get_sync_time(newest=False, debug=debug)
626    max_bound_time_days = STATIC_CONFIG['pipes']['max_bound_time_days']
627
628    extreme_sync_times_delta = (
629        hasattr(oldest_sync_time, 'tzinfo')
630        and (sync_time - oldest_sync_time) >= timedelta(days=max_bound_time_days)
631    )
632
633    return (
634        bound_time
635        if bound_time > oldest_sync_time or extreme_sync_times_delta
636        else None
637    )

The bound time is the limit at which long-running verification syncs should stop. A value of None means verification syncs should be unbounded.

Like deriving a backtrack time from pipe.get_sync_time(), the bound time is the sync time minus a large window (e.g. 366 days).

Unbound verification syncs (i.e. bound_time is None) if the oldest sync time is less than the bound interval.

Returns
  • A datetime or int corresponding to the
  • begin bound for verification and deduplication syncs.
def delete(self, drop: bool = True, debug: bool = False, **kw) -> Tuple[bool, str]:
12def delete(
13    self,
14    drop: bool = True,
15    debug: bool = False,
16    **kw
17) -> SuccessTuple:
18    """
19    Call the Pipe's instance connector's `delete_pipe()` method.
20
21    Parameters
22    ----------
23    drop: bool, default True
24        If `True`, drop the pipes' target table.
25
26    debug : bool, default False
27        Verbosity toggle.
28
29    Returns
30    -------
31    A `SuccessTuple` of success (`bool`), message (`str`).
32
33    """
34    from meerschaum.utils.warnings import warn
35    from meerschaum.utils.venv import Venv
36    from meerschaum.connectors import get_connector_plugin
37
38    if self.temporary:
39        if self.cache:
40            invalidate_success, invalidate_msg = self._invalidate_cache(hard=True, debug=debug)
41            if not invalidate_success:
42                return invalidate_success, invalidate_msg
43
44        return (
45            False,
46            "Cannot delete pipes created with `temporary=True` (read-only). "
47            + "You may want to call `pipe.drop()` instead."
48        )
49
50    if drop:
51        drop_success, drop_msg = self.drop(debug=debug)
52        if not drop_success:
53            warn(f"Failed to drop {self}:\n{drop_msg}")
54
55    with Venv(get_connector_plugin(self.instance_connector)):
56        result = self.instance_connector.delete_pipe(self, debug=debug, **kw)
57
58    if not isinstance(result, tuple):
59        return False, f"Received an unexpected result from '{self.instance_connector}': {result}"
60
61    if result[0]:
62        self._invalidate_cache(hard=True, debug=debug)
63        self._clear_cache_key('_id', debug=debug)
64
65    return result

Call the Pipe's instance connector's delete_pipe() method.

Parameters
  • drop (bool, default True): If True, drop the pipes' target table.
  • debug (bool, default False): Verbosity toggle.
Returns
def drop(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
14def drop(
15    self,
16    debug: bool = False,
17    **kw: Any
18) -> SuccessTuple:
19    """
20    Call the Pipe's instance connector's `drop_pipe()` method.
21
22    Parameters
23    ----------
24    debug: bool, default False:
25        Verbosity toggle.
26
27    Returns
28    -------
29    A `SuccessTuple` of success, message.
30
31    """
32    from meerschaum.utils.venv import Venv
33    from meerschaum.connectors import get_connector_plugin
34
35    self._clear_cache_key('_exists', debug=debug)
36
37    with Venv(get_connector_plugin(self.instance_connector)):
38        if hasattr(self.instance_connector, 'drop_pipe'):
39            result = self.instance_connector.drop_pipe(self, debug=debug, **kw)
40        else:
41            result = (
42                False,
43                (
44                    "Cannot drop pipes for instance connectors of type "
45                    f"'{self.instance_connector.type}'."
46                )
47            )
48
49    self._clear_cache_key('_exists', debug=debug)
50    self._clear_cache_key('_exists_timestamp', debug=debug)
51
52    return result

Call the Pipe's instance connector's drop_pipe() method.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
def drop_indices( self, columns: Optional[List[str]] = None, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
 55def drop_indices(
 56    self,
 57    columns: Optional[List[str]] = None,
 58    debug: bool = False,
 59    **kw: Any
 60) -> SuccessTuple:
 61    """
 62    Call the Pipe's instance connector's `drop_indices()` method.
 63
 64    Parameters
 65    ----------
 66    columns: Optional[List[str]] = None
 67        If provided, only drop indices in the given list.
 68
 69    debug: bool, default False:
 70        Verbosity toggle.
 71
 72    Returns
 73    -------
 74    A `SuccessTuple` of success, message.
 75
 76    """
 77    from meerschaum.utils.venv import Venv
 78    from meerschaum.connectors import get_connector_plugin
 79
 80    self._clear_cache_key('_columns_indices', debug=debug)
 81    self._clear_cache_key('_columns_indices_timestamp', debug=debug)
 82    self._clear_cache_key('_columns_types', debug=debug)
 83    self._clear_cache_key('_columns_types_timestamp', debug=debug)
 84
 85    with Venv(get_connector_plugin(self.instance_connector)):
 86        if hasattr(self.instance_connector, 'drop_pipe_indices'):
 87            result = self.instance_connector.drop_pipe_indices(
 88                self,
 89                columns=columns,
 90                debug=debug,
 91                **kw
 92            )
 93        else:
 94            result = (
 95                False,
 96                (
 97                    "Cannot drop indices for instance connectors of type "
 98                    f"'{self.instance_connector.type}'."
 99                )
100            )
101
102    self._clear_cache_key('_columns_indices', debug=debug)
103    self._clear_cache_key('_columns_indices_timestamp', debug=debug)
104    self._clear_cache_key('_columns_types', debug=debug)
105    self._clear_cache_key('_columns_types_timestamp', debug=debug)
106
107    return result

Call the Pipe's instance connector's drop_indices() method.

Parameters
  • columns (Optional[List[str]] = None): If provided, only drop indices in the given list.
  • debug (bool, default False:): Verbosity toggle.
Returns
def compress(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
14def compress(
15    self,
16    debug: bool = False,
17    **kw: Any
18) -> SuccessTuple:
19    """
20    Call the Pipe's instance connector's `compress_pipe()` method.
21
22    For TimescaleDB hypertables this enables and applies native compression.
23    Other flavors fall back to their respective compression mechanisms where supported.
24
25    Parameters
26    ----------
27    debug: bool, default False:
28        Verbosity toggle.
29
30    Returns
31    -------
32    A `SuccessTuple` of success, message.
33    """
34    from meerschaum.utils.venv import Venv
35    from meerschaum.connectors import get_connector_plugin
36
37    try:
38        with Venv(get_connector_plugin(self.instance_connector)):
39            if hasattr(self.instance_connector, 'compress_pipe'):
40                result = self.instance_connector.compress_pipe(self, debug=debug, **kw)
41            else:
42                result = (
43                    False,
44                    (
45                        "Cannot compress pipes for instance connectors of type "
46                        f"'{self.instance_connector.type}'."
47                    )
48                )
49    except NotImplementedError:
50        result = (
51            False,
52            (
53                "Compression is not implemented for instance connectors of type "
54                f"'{self.instance_connector.type}'."
55            )
56        )
57
58    self._clear_cache_key('_exists', debug=debug)
59    return result

Call the Pipe's instance connector's compress_pipe() method.

For TimescaleDB hypertables this enables and applies native compression. Other flavors fall back to their respective compression mechanisms where supported.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
def decompress(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
 62def decompress(
 63    self,
 64    debug: bool = False,
 65    **kw: Any
 66) -> SuccessTuple:
 67    """
 68    Call the Pipe's instance connector's `decompress_pipe()` method, the inverse of `compress()`.
 69
 70    For TimescaleDB hypertables this removes the compression policy and converts compressed
 71    chunks back to row-store. Other flavors fall back to their respective mechanisms where
 72    supported.
 73
 74    Parameters
 75    ----------
 76    debug: bool, default False:
 77        Verbosity toggle.
 78
 79    Returns
 80    -------
 81    A `SuccessTuple` of success, message.
 82    """
 83    from meerschaum.utils.venv import Venv
 84    from meerschaum.connectors import get_connector_plugin
 85
 86    try:
 87        with Venv(get_connector_plugin(self.instance_connector)):
 88            if hasattr(self.instance_connector, 'decompress_pipe'):
 89                result = self.instance_connector.decompress_pipe(self, debug=debug, **kw)
 90            else:
 91                result = (
 92                    False,
 93                    (
 94                        "Cannot decompress pipes for instance connectors of type "
 95                        f"'{self.instance_connector.type}'."
 96                    )
 97                )
 98    except NotImplementedError:
 99        result = (
100            False,
101            (
102                "Decompression is not implemented for instance connectors of type "
103                f"'{self.instance_connector.type}'."
104            )
105        )
106
107    self._clear_cache_key('_exists', debug=debug)
108    return result

Call the Pipe's instance connector's decompress_pipe() method, the inverse of compress().

For TimescaleDB hypertables this removes the compression policy and converts compressed chunks back to row-store. Other flavors fall back to their respective mechanisms where supported.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
def vacuum( self, full: bool = False, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
14def vacuum(
15    self,
16    full: bool = False,
17    debug: bool = False,
18    **kw: Any
19) -> SuccessTuple:
20    """
21    Call the Pipe's instance connector's `vacuum_pipe()` method to reclaim disk space.
22
23    For PostgreSQL-family tables this runs `VACUUM` (optionally `VACUUM FULL`); other flavors
24    fall back to their respective space-reclaiming mechanisms where supported.
25
26    Parameters
27    ----------
28    full: bool, default False
29        If `True` (PostgreSQL family only), run `VACUUM FULL` to return freed space to the OS.
30
31    debug: bool, default False
32        Verbosity toggle.
33
34    Returns
35    -------
36    A `SuccessTuple` of success, message.
37    """
38    from meerschaum.utils.venv import Venv
39    from meerschaum.connectors import get_connector_plugin
40
41    try:
42        with Venv(get_connector_plugin(self.instance_connector)):
43            if hasattr(self.instance_connector, 'vacuum_pipe'):
44                result = self.instance_connector.vacuum_pipe(self, full=full, debug=debug, **kw)
45            else:
46                result = (
47                    False,
48                    (
49                        "Cannot vacuum pipes for instance connectors of type "
50                        f"'{self.instance_connector.type}'."
51                    )
52                )
53    except NotImplementedError:
54        result = (
55            False,
56            (
57                "Vacuuming is not implemented for instance connectors of type "
58                f"'{self.instance_connector.type}'."
59            )
60        )
61
62    self._clear_cache_key('_exists', debug=debug)
63    return result

Call the Pipe's instance connector's vacuum_pipe() method to reclaim disk space.

For PostgreSQL-family tables this runs VACUUM (optionally VACUUM FULL); other flavors fall back to their respective space-reclaiming mechanisms where supported.

Parameters
  • full (bool, default False): If True (PostgreSQL family only), run VACUUM FULL to return freed space to the OS.
  • debug (bool, default False): Verbosity toggle.
Returns
def analyze(self, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
 66def analyze(
 67    self,
 68    debug: bool = False,
 69    **kw: Any
 70) -> SuccessTuple:
 71    """
 72    Call the Pipe's instance connector's `analyze_pipe()` method to refresh planner statistics.
 73
 74    Parameters
 75    ----------
 76    debug: bool, default False
 77        Verbosity toggle.
 78
 79    Returns
 80    -------
 81    A `SuccessTuple` of success, message.
 82    """
 83    from meerschaum.utils.venv import Venv
 84    from meerschaum.connectors import get_connector_plugin
 85
 86    try:
 87        with Venv(get_connector_plugin(self.instance_connector)):
 88            if hasattr(self.instance_connector, 'analyze_pipe'):
 89                result = self.instance_connector.analyze_pipe(self, debug=debug, **kw)
 90            else:
 91                result = (
 92                    False,
 93                    (
 94                        "Cannot analyze pipes for instance connectors of type "
 95                        f"'{self.instance_connector.type}'."
 96                    )
 97                )
 98    except NotImplementedError:
 99        result = (
100            False,
101            (
102                "Analyzing is not implemented for instance connectors of type "
103                f"'{self.instance_connector.type}'."
104            )
105        )
106
107    return result

Call the Pipe's instance connector's analyze_pipe() method to refresh planner statistics.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
def repartition( self, chunk_minutes: Optional[int] = None, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
110def repartition(
111    self,
112    chunk_minutes: Optional[int] = None,
113    debug: bool = False,
114    **kw: Any
115) -> SuccessTuple:
116    """
117    Call the Pipe's instance connector's `partition_pipe()` method to rebuild the target table
118    to a new partition (chunk) width.
119
120    On TimescaleDB this changes the chunk interval for future chunks. On PostgreSQL / PostGIS,
121    MySQL / MariaDB, and MSSQL it rebuilds the natively range-partitioned table at the new width.
122
123    Parameters
124    ----------
125    chunk_minutes: Optional[int], default None
126        The new partition width in minutes. Defaults to the pipe's `verify.chunk_minutes`.
127
128    debug: bool, default False
129        Verbosity toggle.
130
131    Returns
132    -------
133    A `SuccessTuple` of success, message.
134    """
135    from meerschaum.utils.venv import Venv
136    from meerschaum.connectors import get_connector_plugin
137
138    try:
139        with Venv(get_connector_plugin(self.instance_connector)):
140            if hasattr(self.instance_connector, 'partition_pipe'):
141                result = self.instance_connector.partition_pipe(
142                    self, chunk_minutes=chunk_minutes, debug=debug, **kw
143                )
144            else:
145                result = (
146                    False,
147                    (
148                        "Cannot repartition pipes for instance connectors of type "
149                        f"'{self.instance_connector.type}'."
150                    )
151                )
152    except NotImplementedError:
153        result = (
154            False,
155            (
156                "Repartitioning is not implemented for instance connectors of type "
157                f"'{self.instance_connector.type}'."
158            )
159        )
160
161    self._clear_cache_key('_exists', debug=debug)
162    return result

Call the Pipe's instance connector's partition_pipe() method to rebuild the target table to a new partition (chunk) width.

On TimescaleDB this changes the chunk interval for future chunks. On PostgreSQL / PostGIS, MySQL / MariaDB, and MSSQL it rebuilds the natively range-partitioned table at the new width.

Parameters
  • chunk_minutes (Optional[int], default None): The new partition width in minutes. Defaults to the pipe's verify.chunk_minutes.
  • debug (bool, default False): Verbosity toggle.
Returns
def create_indices( self, columns: Optional[List[str]] = None, debug: bool = False, **kw: Any) -> Tuple[bool, str]:
14def create_indices(
15    self,
16    columns: Optional[List[str]] = None,
17    debug: bool = False,
18    **kw: Any
19) -> SuccessTuple:
20    """
21    Call the Pipe's instance connector's `create_pipe_indices()` method.
22
23    Parameters
24    ----------
25    debug: bool, default False:
26        Verbosity toggle.
27
28    Returns
29    -------
30    A `SuccessTuple` of success, message.
31
32    """
33    from meerschaum.utils.venv import Venv
34    from meerschaum.connectors import get_connector_plugin
35
36    self._clear_cache_key('_columns_indices', debug=debug)
37    self._clear_cache_key('_columns_indices_timestamp', debug=debug)
38    self._clear_cache_key('_columns_types', debug=debug)
39    self._clear_cache_key('_columns_types_timestamp', debug=debug)
40
41    with Venv(get_connector_plugin(self.instance_connector)):
42        if hasattr(self.instance_connector, 'create_pipe_indices'):
43            result = self.instance_connector.create_pipe_indices(
44                self,
45                columns=columns,
46                debug=debug,
47                **kw
48            )
49        else:
50            result = (
51                False,
52                (
53                    "Cannot create indices for instance connectors of type "
54                    f"'{self.instance_connector.type}'."
55                )
56            )
57
58    self._clear_cache_key('_columns_indices', debug=debug)
59    self._clear_cache_key('_columns_indices_timestamp', debug=debug)
60    self._clear_cache_key('_columns_types', debug=debug)
61    self._clear_cache_key('_columns_types_timestamp', debug=debug)
62
63    return result

Call the Pipe's instance connector's create_pipe_indices() method.

Parameters
  • debug (bool, default False:): Verbosity toggle.
Returns
def clear( self, begin: Optional[datetime.datetime] = None, end: Optional[datetime.datetime] = None, params: Optional[Dict[str, Any]] = None, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
16def clear(
17    self,
18    begin: Optional[datetime] = None,
19    end: Optional[datetime] = None,
20    params: Optional[Dict[str, Any]] = None,
21    debug: bool = False,
22    **kwargs: Any
23) -> SuccessTuple:
24    """
25    Call the Pipe's instance connector's `clear_pipe` method.
26
27    Parameters
28    ----------
29    begin: Optional[datetime], default None:
30        If provided, only remove rows newer than this datetime value.
31
32    end: Optional[datetime], default None:
33        If provided, only remove rows older than this datetime column (not including end).
34
35    params: Optional[Dict[str, Any]], default None
36         See `meerschaum.utils.sql.build_where`.
37
38    debug: bool, default False:
39        Verbositity toggle.
40
41    Returns
42    -------
43    A `SuccessTuple` corresponding to whether this procedure completed successfully.
44
45    Examples
46    --------
47    >>> pipe = mrsm.Pipe('test', 'test', columns={'datetime': 'dt'}, instance='sql:local')
48    >>> pipe.sync({'dt': [datetime(2020, 1, 1, 0, 0)]})
49    >>> pipe.sync({'dt': [datetime(2021, 1, 1, 0, 0)]})
50    >>> pipe.sync({'dt': [datetime(2022, 1, 1, 0, 0)]})
51    >>> 
52    >>> pipe.clear(begin=datetime(2021, 1, 1, 0, 0))
53    >>> pipe.get_data()
54              dt
55    0 2020-01-01
56
57    """
58    from meerschaum.utils.warnings import warn
59    from meerschaum.utils.venv import Venv
60    from meerschaum.connectors import get_connector_plugin
61
62    begin, end = self.parse_date_bounds(begin, end)
63
64    with Venv(get_connector_plugin(self.instance_connector)):
65        return self.instance_connector.clear_pipe(
66            self,
67            begin=begin,
68            end=end,
69            params=params,
70            debug=debug,
71            **kwargs
72        )

Call the Pipe's instance connector's clear_pipe method.

Parameters
  • begin (Optional[datetime], default None:): If provided, only remove rows newer than this datetime value.
  • end (Optional[datetime], default None:): If provided, only remove rows older than this datetime column (not including end).
  • params (Optional[Dict[str, Any]], default None): See meerschaum.utils.sql.build_where.
  • debug (bool, default False:): Verbositity toggle.
Returns
  • A SuccessTuple corresponding to whether this procedure completed successfully.
Examples
>>> pipe = mrsm.Pipe('test', 'test', columns={'datetime': 'dt'}, instance='sql:local')
>>> pipe.sync({'dt': [datetime(2020, 1, 1, 0, 0)]})
>>> pipe.sync({'dt': [datetime(2021, 1, 1, 0, 0)]})
>>> pipe.sync({'dt': [datetime(2022, 1, 1, 0, 0)]})
>>> 
>>> pipe.clear(begin=datetime(2021, 1, 1, 0, 0))
>>> pipe.get_data()
          dt
0 2020-01-01
def deduplicate( self, begin: Union[datetime.datetime, int, NoneType] = None, end: Union[datetime.datetime, int, NoneType] = None, params: Optional[Dict[str, Any]] = None, chunk_interval: Union[datetime.datetime, int, NoneType] = None, bounded: Optional[bool] = None, workers: Optional[int] = None, debug: bool = False, _use_instance_method: bool = True, **kwargs: Any) -> Tuple[bool, str]:
 15def deduplicate(
 16    self,
 17    begin: Union[datetime, int, None] = None,
 18    end: Union[datetime, int, None] = None,
 19    params: Optional[Dict[str, Any]] = None,
 20    chunk_interval: Union[datetime, int, None] = None,
 21    bounded: Optional[bool] = None,
 22    workers: Optional[int] = None,
 23    debug: bool = False,
 24    _use_instance_method: bool = True,
 25    **kwargs: Any
 26) -> SuccessTuple:
 27    """
 28    Call the Pipe's instance connector's `delete_duplicates` method to delete duplicate rows.
 29
 30    Parameters
 31    ----------
 32    begin: Union[datetime, int, None], default None:
 33        If provided, only deduplicate rows newer than this datetime value.
 34
 35    end: Union[datetime, int, None], default None:
 36        If provided, only deduplicate rows older than this datetime column (not including end).
 37
 38    params: Optional[Dict[str, Any]], default None
 39        Restrict deduplication to this filter (for multiplexed data streams).
 40        See `meerschaum.utils.sql.build_where`.
 41
 42    chunk_interval: Union[timedelta, int, None], default None
 43        If provided, use this for the chunk bounds.
 44        Defaults to the value set in `pipe.parameters['verify']['chunk_minutes']` (43200 — 30 days).
 45
 46    bounded: Optional[bool], default None
 47        Only check outside the oldest and newest sync times if bounded is explicitly `False`.
 48
 49    workers: Optional[int], default None
 50        If the instance connector is thread-safe, limit concurrenct syncs to this many threads.
 51
 52    debug: bool, default False:
 53        Verbositity toggle.
 54
 55    kwargs: Any
 56        All other keyword arguments are passed to
 57        `pipe.sync()`, `pipe.clear()`, and `pipe.get_data().
 58
 59    Returns
 60    -------
 61    A `SuccessTuple` corresponding to whether all of the chunks were successfully deduplicated.
 62    """
 63    from meerschaum.utils.warnings import warn, info
 64    from meerschaum.utils.misc import interval_str, items_str
 65    from meerschaum.utils.venv import Venv
 66    from meerschaum.connectors import get_connector_plugin
 67    from meerschaum.utils.pool import get_pool
 68
 69    begin, end = self.parse_date_bounds(begin, end)
 70
 71    workers = self.get_num_workers(workers=workers)
 72    pool = get_pool(workers=workers)
 73
 74    if _use_instance_method:
 75        with Venv(get_connector_plugin(self.instance_connector)):
 76            if hasattr(self.instance_connector, 'deduplicate_pipe'):
 77                return self.instance_connector.deduplicate_pipe(
 78                    self,
 79                    begin=begin,
 80                    end=end,
 81                    params=params,
 82                    bounded=bounded,
 83                    debug=debug,
 84                    **kwargs
 85                )
 86
 87    ### Only unbound if explicitly False.
 88    if bounded is None:
 89        bounded = True
 90    chunk_interval = self.get_chunk_interval(chunk_interval, debug=debug)
 91
 92    bound_time = self.get_bound_time(debug=debug)
 93    if bounded and begin is None:
 94        begin = (
 95            bound_time
 96            if bound_time is not None
 97            else self.get_sync_time(newest=False, debug=debug)
 98        )
 99    if bounded and end is None:
100        end = self.get_sync_time(newest=True, debug=debug)
101        if end is not None:
102            end += (
103                timedelta(minutes=1)
104                if hasattr(end, 'tzinfo')
105                else 1
106            )
107
108    chunk_bounds = self.get_chunk_bounds(
109        bounded=bounded,
110        begin=begin,
111        end=end,
112        chunk_interval=chunk_interval,
113        debug=debug,
114    )
115
116    indices = [col for col in self.columns.values() if col]
117    if not indices:
118        return False, "Cannot deduplicate without index columns."
119
120    def process_chunk_bounds(bounds) -> Tuple[
121        Tuple[
122            Union[datetime, int, None],
123            Union[datetime, int, None]
124        ],
125        SuccessTuple
126    ]:
127        ### Only selecting the index values here to keep bandwidth down.
128        chunk_begin, chunk_end = bounds
129        chunk_df = self.get_data(
130            select_columns=indices, 
131            begin=chunk_begin,
132            end=chunk_end,
133            params=params,
134            debug=debug,
135        )
136        if chunk_df is None:
137            return bounds, (True, "")
138        existing_chunk_len = len(chunk_df)
139        deduped_chunk_df = chunk_df.drop_duplicates(keep='last')
140        deduped_chunk_len = len(deduped_chunk_df)
141
142        if existing_chunk_len == deduped_chunk_len:
143            return bounds, (True, "")
144
145        chunk_msg_header = f"\n{chunk_begin} - {chunk_end}"
146        chunk_msg_body = ""
147
148        full_chunk = self.get_data(
149            begin=chunk_begin,
150            end=chunk_end,
151            params=params,
152            debug=debug,
153        )
154        if full_chunk is None or len(full_chunk) == 0:
155            return bounds, (True, f"{chunk_msg_header}\nChunk is empty, skipping...")
156
157        chunk_indices = [ix for ix in indices if ix in full_chunk.columns]
158        if not chunk_indices:
159            return bounds, (False, f"None of {items_str(indices)} were present in chunk.")
160        try:
161            full_chunk = full_chunk.drop_duplicates(
162                subset=chunk_indices,
163                keep='last'
164            ).reset_index(
165                drop=True,
166            )
167        except Exception as e:
168            return (
169                bounds,
170                (False, f"Failed to deduplicate chunk on {items_str(chunk_indices)}:\n({e})")
171            )
172
173        clear_success, clear_msg = self.clear(
174            begin=chunk_begin,
175            end=chunk_end,
176            params=params,
177            debug=debug,
178        )
179        if not clear_success:
180            chunk_msg_body += f"Failed to clear chunk while deduplicating:\n{clear_msg}\n"
181            warn(chunk_msg_body)
182
183        sync_success, sync_msg = self.sync(full_chunk, debug=debug)
184        if not sync_success:
185            chunk_msg_body += f"Failed to sync chunk while deduplicating:\n{sync_msg}\n"
186
187        ### Finally check if the deduplication worked.
188        chunk_rowcount = self.get_rowcount(
189            begin=chunk_begin,
190            end=chunk_end,
191            params=params,
192            debug=debug,
193        )
194        if chunk_rowcount != deduped_chunk_len:
195            return bounds, (
196                False, (
197                    chunk_msg_header + "\n"
198                    + chunk_msg_body + ("\n" if chunk_msg_body else '')
199                    + "Chunk rowcounts still differ ("
200                    + f"{chunk_rowcount} rowcount vs {deduped_chunk_len} chunk length)."
201                )
202            )
203
204        return bounds, (
205            True, (
206                chunk_msg_header + "\n"
207                + chunk_msg_body + ("\n" if chunk_msg_body else '')
208                + f"Deduplicated chunk from {existing_chunk_len} to {chunk_rowcount} rows."
209            )
210        )
211
212    info(
213        f"Deduplicating {len(chunk_bounds)} chunk"
214        + ('s' if len(chunk_bounds) != 1 else '')
215        + f" ({'un' if not bounded else ''}bounded)"
216        + f" of size '{interval_str(chunk_interval)}'"
217        + f" on {self}."
218    )
219    bounds_success_tuples = dict(pool.map(process_chunk_bounds, chunk_bounds))
220    bounds_successes = {
221        bounds: success_tuple
222        for bounds, success_tuple in bounds_success_tuples.items()
223        if success_tuple[0]
224    }
225    bounds_failures = {
226        bounds: success_tuple
227        for bounds, success_tuple in bounds_success_tuples.items()
228        if not success_tuple[0]
229    }
230
231    ### No need to retry if everything failed.
232    if len(bounds_failures) > 0 and len(bounds_successes) == 0:
233        return (
234            False,
235            (
236                f"Failed to deduplicate {len(bounds_failures)} chunk"
237                + ('s' if len(bounds_failures) != 1 else '')
238                + ".\n"
239                + "\n".join([msg for _, (_, msg) in bounds_failures.items() if msg])
240            )
241        )
242
243    retry_bounds = [bounds for bounds in bounds_failures]
244    if not retry_bounds:
245        return (
246            True,
247            (
248                f"Successfully deduplicated {len(bounds_successes)} chunk"
249                + ('s' if len(bounds_successes) != 1 else '')
250                + ".\n"
251                + "\n".join([msg for _, (_, msg) in bounds_successes.items() if msg])
252            ).rstrip('\n')
253        )
254
255    info(f"Retrying {len(retry_bounds)} chunks for {self}...")
256    retry_bounds_success_tuples = dict(pool.map(process_chunk_bounds, retry_bounds))
257    retry_bounds_successes = {
258        bounds: success_tuple
259        for bounds, success_tuple in retry_bounds_success_tuples.items()
260        if success_tuple[0]
261    }
262    retry_bounds_failures = {
263        bounds: success_tuple
264        for bounds, success_tuple in retry_bounds_success_tuples.items()
265        if not success_tuple[0]
266    }
267
268    bounds_successes.update(retry_bounds_successes)
269    if not retry_bounds_failures:
270        return (
271            True,
272            (
273                f"Successfully deduplicated {len(bounds_successes)} chunk"
274                + ('s' if len(bounds_successes) != 1 else '')
275                + f"({len(retry_bounds_successes)} retried):\n"
276                + "\n".join([msg for _, (_, msg) in bounds_successes.items() if msg])
277            ).rstrip('\n')
278        )
279
280    return (
281        False,
282        (
283            f"Failed to deduplicate {len(bounds_failures)} chunk"
284            + ('s' if len(retry_bounds_failures) != 1 else '')
285            + ".\n"
286            + "\n".join([msg for _, (_, msg) in retry_bounds_failures.items() if msg])
287        ).rstrip('\n')
288    )

Call the Pipe's instance connector's delete_duplicates method to delete duplicate rows.

Parameters
  • begin (Union[datetime, int, None], default None:): If provided, only deduplicate rows newer than this datetime value.
  • end (Union[datetime, int, None], default None:): If provided, only deduplicate rows older than this datetime column (not including end).
  • params (Optional[Dict[str, Any]], default None): Restrict deduplication to this filter (for multiplexed data streams). See meerschaum.utils.sql.build_where.
  • chunk_interval (Union[timedelta, int, None], default None): If provided, use this for the chunk bounds. Defaults to the value set in pipe.parameters['verify']['chunk_minutes'] (43200 — 30 days).
  • bounded (Optional[bool], default None): Only check outside the oldest and newest sync times if bounded is explicitly False.
  • workers (Optional[int], default None): If the instance connector is thread-safe, limit concurrenct syncs to this many threads.
  • debug (bool, default False:): Verbositity toggle.
  • kwargs (Any): All other keyword arguments are passed to pipe.sync(), pipe.clear(), and `pipe.get_data().
Returns
  • A SuccessTuple corresponding to whether all of the chunks were successfully deduplicated.
def bootstrap( self, debug: bool = False, yes: bool = False, force: bool = False, noask: bool = False, shell: bool = False, **kw) -> Tuple[bool, str]:
 16def bootstrap(
 17    self,
 18    debug: bool = False,
 19    yes: bool = False,
 20    force: bool = False,
 21    noask: bool = False,
 22    shell: bool = False,
 23    **kw
 24) -> SuccessTuple:
 25    """
 26    Prompt the user to create a pipe's requirements all from one method.
 27    This method shouldn't be used in any automated scripts because it interactively
 28    prompts the user and therefore may hang.
 29
 30    Parameters
 31    ----------
 32    debug: bool, default False:
 33        Verbosity toggle.
 34
 35    yes: bool, default False:
 36        Print the questions and automatically agree.
 37
 38    force: bool, default False:
 39        Skip the questions and agree anyway.
 40
 41    noask: bool, default False:
 42        Print the questions but go with the default answer.
 43
 44    shell: bool, default False:
 45        Used to determine if we are in the interactive shell.
 46        
 47    Returns
 48    -------
 49    A `SuccessTuple` corresponding to the success of this procedure.
 50
 51    """
 52
 53    from meerschaum.utils.warnings import info
 54    from meerschaum.utils.prompt import prompt, yes_no
 55    from meerschaum.utils.formatting import pprint
 56    from meerschaum.config import get_config
 57    from meerschaum.utils.formatting._shell import clear_screen
 58    from meerschaum.utils.formatting import print_tuple
 59    from meerschaum.actions import actions
 60    from meerschaum.utils.venv import Venv
 61    from meerschaum.connectors import get_connector_plugin
 62
 63    _clear = get_config('shell', 'clear_screen', patch=True)
 64
 65    if self.id is not None:
 66        delete_tuple = self.delete(debug=debug)
 67        if not delete_tuple[0]:
 68            return delete_tuple
 69
 70    if _clear:
 71        clear_screen(debug=debug)
 72
 73    _parameters = _get_parameters(self, debug=debug)
 74    self.parameters = _parameters
 75    pprint(self.parameters)
 76    try:
 77        prompt(
 78            f"\n    Press [Enter] to register {self} with the above configuration:",
 79            icon = False
 80        )
 81    except KeyboardInterrupt:
 82        return False, f"Aborted bootstrapping {self}."
 83
 84    with Venv(get_connector_plugin(self.instance_connector)):
 85        register_tuple = self.instance_connector.register_pipe(self, debug=debug)
 86
 87    if not register_tuple[0]:
 88        return register_tuple
 89
 90    if _clear:
 91        clear_screen(debug=debug)
 92
 93    try:
 94        if yes_no(
 95            f"Would you like to edit the definition for {self}?",
 96            yes=yes,
 97            noask=noask,
 98            default='n',
 99        ):
100            edit_tuple = self.edit_definition(debug=debug)
101            if not edit_tuple[0]:
102                return edit_tuple
103
104        if yes_no(
105            f"Would you like to try syncing {self} now?",
106            yes=yes,
107            noask=noask,
108            default='n',
109        ):
110            sync_tuple = actions['sync'](
111                ['pipes'],
112                connector_keys=[self.connector_keys],
113                metric_keys=[self.metric_key],
114                location_keys=[self.location_key],
115                mrsm_instance=str(self.instance_connector),
116                debug=debug,
117                shell=shell,
118            )
119            if not sync_tuple[0]:
120                return sync_tuple
121    except Exception as e:
122        return False, f"Failed to bootstrap {self}:\n" + str(e)
123
124    print_tuple((True, f"Finished bootstrapping {self}!"))
125    info(
126        "You can edit this pipe later with `edit pipes` "
127        + "or set the definition with `edit pipes definition`.\n"
128        + "    To sync data into your pipe, run `sync pipes`."
129    )
130
131    return True, "Success"

Prompt the user to create a pipe's requirements all from one method. This method shouldn't be used in any automated scripts because it interactively prompts the user and therefore may hang.

Parameters
  • debug (bool, default False:): Verbosity toggle.
  • yes (bool, default False:): Print the questions and automatically agree.
  • force (bool, default False:): Skip the questions and agree anyway.
  • noask (bool, default False:): Print the questions but go with the default answer.
  • shell (bool, default False:): Used to determine if we are in the interactive shell.
Returns
  • A SuccessTuple corresponding to the success of this procedure.
def enforce_dtypes( self, df: pandas.DataFrame, chunksize: Optional[int] = -1, enforce: bool = True, safe_copy: bool = True, dtypes: Optional[Dict[str, str]] = None, as_polars: bool = False, debug: bool = False) -> pandas.DataFrame:
 20def enforce_dtypes(
 21    self,
 22    df: 'pd.DataFrame',
 23    chunksize: Optional[int] = -1,
 24    enforce: bool = True,
 25    safe_copy: bool = True,
 26    dtypes: Optional[Dict[str, str]] = None,
 27    as_polars: bool = False,
 28    debug: bool = False,
 29) -> 'pd.DataFrame':
 30    """
 31    Cast the input dataframe to the pipe's registered data types.
 32    If the pipe does not exist and dtypes are not set, return the dataframe.
 33    """
 34    import traceback
 35    from meerschaum.utils.warnings import warn
 36    from meerschaum.utils.debug import dprint
 37    from meerschaum.utils.dataframe import (
 38        parse_df_datetimes,
 39        enforce_dtypes as _enforce_dtypes,
 40        parse_simple_lines,
 41        to_pandas,
 42        to_polars,
 43    )
 44    from meerschaum.utils.dtypes import are_dtypes_equal
 45    from meerschaum.utils.packages import import_pandas
 46    pd = import_pandas(debug=debug)
 47    if df is None:
 48        if debug:
 49            dprint(
 50                "Received None instead of a DataFrame.\n"
 51                + "    Skipping dtype enforcement..."
 52            )
 53        return df
 54
 55    if not self.enforce:
 56        enforce = False
 57
 58    explicit_dtypes = self.get_dtypes(infer=False, debug=debug) if enforce else {}
 59    pipe_dtypes = self.get_dtypes(infer=True, debug=debug) if not dtypes else dtypes
 60
 61    try:
 62        if isinstance(df, str):
 63            if df.strip() and df.strip()[0] not in ('{', '['):
 64                df = parse_df_datetimes(
 65                    parse_simple_lines(df),
 66                    ignore_cols=[
 67                        col
 68                        for col, dtype in pipe_dtypes.items()
 69                        if (not enforce or not are_dtypes_equal(dtype, 'datetime'))
 70                    ],
 71                )
 72            else:
 73                df = parse_df_datetimes(
 74                    pd.read_json(StringIO(df)),
 75                    ignore_cols=[
 76                        col
 77                        for col, dtype in pipe_dtypes.items()
 78                        if (not enforce or not are_dtypes_equal(dtype, 'datetime'))
 79                    ],
 80                    ignore_all=(not enforce),
 81                    strip_timezone=(self.tzinfo is None),
 82                    chunksize=chunksize,
 83                    debug=debug,
 84                )
 85        elif isinstance(df, (dict, list, tuple)):
 86            df = parse_df_datetimes(
 87                df,
 88                ignore_cols=[
 89                    col
 90                    for col, dtype in pipe_dtypes.items()
 91                    if (not enforce or not are_dtypes_equal(str(dtype), 'datetime'))
 92                ],
 93                strip_timezone=(self.tzinfo is None),
 94                chunksize=chunksize,
 95                debug=debug,
 96            )
 97    except Exception as e:
 98        warn(f"Unable to cast incoming data as a DataFrame...:\n{e}\n\n{traceback.format_exc()}")
 99        return None
100
101    if not pipe_dtypes:
102        if debug:
103            dprint(
104                f"Could not find dtypes for {self}.\n"
105                + "Skipping dtype enforcement..."
106            )
107        return to_polars(df) if as_polars else to_pandas(df)
108
109    return _enforce_dtypes(
110        df,
111        pipe_dtypes,
112        explicit_dtypes=explicit_dtypes,
113        safe_copy=safe_copy,
114        strip_timezone=(self.tzinfo is None),
115        coerce_numeric=self.mixed_numerics,
116        coerce_timezone=enforce,
117        as_polars=as_polars,
118        debug=debug,
119    )

Cast the input dataframe to the pipe's registered data types. If the pipe does not exist and dtypes are not set, return the dataframe.

def infer_dtypes( self, persist: bool = False, refresh: bool = False, debug: bool = False) -> Dict[str, Any]:
122def infer_dtypes(
123    self,
124    persist: bool = False,
125    refresh: bool = False,
126    debug: bool = False,
127) -> Dict[str, Any]:
128    """
129    If `dtypes` is not set in `meerschaum.Pipe.parameters`,
130    infer the data types from the underlying table if it exists.
131
132    Parameters
133    ----------
134    persist: bool, default False
135        If `True`, persist the inferred data types to `meerschaum.Pipe.parameters`.
136        NOTE: Use with caution! Generally `dtypes` is meant to be user-configurable only.
137
138    refresh: bool, default False
139        If `True`, retrieve the latest columns-types for the pipe.
140        See `Pipe.get_columns.types()`.
141
142    Returns
143    -------
144    A dictionary of strings containing the pandas data types for this Pipe.
145    """
146    if not self.exists(debug=debug):
147        return {}
148
149    from meerschaum.utils.dtypes.sql import get_pd_type_from_db_type
150    from meerschaum.utils.dtypes import to_pandas_dtype
151
152    ### NOTE: get_columns_types() may return either the types as
153    ###       PostgreSQL- or Pandas-style.
154    columns_types = self.get_columns_types(refresh=refresh, debug=debug)
155
156    remote_pd_dtypes = {
157        c: (
158            get_pd_type_from_db_type(t, allow_custom_dtypes=True)
159            if str(t).isupper()
160            else to_pandas_dtype(t)
161        )
162        for c, t in columns_types.items()
163    } if columns_types else {}
164    if not persist:
165        return remote_pd_dtypes
166
167    parameters = self.get_parameters(refresh=refresh, debug=debug)
168    dtypes = parameters.get('dtypes', {})
169    dtypes.update({
170        col: typ
171        for col, typ in remote_pd_dtypes.items()
172        if col not in dtypes
173    })
174    self.dtypes = dtypes
175    self.edit(interactive=False, debug=debug)
176    return remote_pd_dtypes

If dtypes is not set in meerschaum.Pipe.parameters, infer the data types from the underlying table if it exists.

Parameters
  • persist (bool, default False): If True, persist the inferred data types to meerschaum.Pipe.parameters. NOTE: Use with caution! Generally dtypes is meant to be user-configurable only.
  • refresh (bool, default False): If True, retrieve the latest columns-types for the pipe. See Pipe.get_columns.types().
Returns
  • A dictionary of strings containing the pandas data types for this Pipe.
def copy_to( self, instance_keys: str, sync: bool = True, begin: Union[datetime.datetime, int, NoneType] = None, end: Union[datetime.datetime, int, NoneType] = None, params: Optional[Dict[str, Any]] = None, chunk_interval: Union[datetime.timedelta, int, NoneType] = None, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
 15def copy_to(
 16    self,
 17    instance_keys: str,
 18    sync: bool = True,
 19    begin: Union[datetime, int, None] = None,
 20    end: Union[datetime, int, None] = None,
 21    params: Optional[Dict[str, Any]] = None,
 22    chunk_interval: Union[timedelta, int, None] = None,
 23    debug: bool = False,
 24    **kwargs: Any
 25) -> SuccessTuple:
 26    """
 27    Copy a pipe to another instance.
 28
 29    Parameters
 30    ----------
 31    instance_keys: str
 32        The instance to which to copy this pipe.
 33
 34    sync: bool, default True
 35        If `True`, sync the source pipe's documents 
 36
 37    begin: Union[datetime, int, None], default None
 38        Beginning datetime value to pass to `Pipe.get_data()`.
 39
 40    end: Union[datetime, int, None], default None
 41        End datetime value to pass to `Pipe.get_data()`.
 42
 43    params: Optional[Dict[str, Any]], default None
 44        Parameters filter to pass to `Pipe.get_data()`.
 45
 46    chunk_interval: Union[timedelta, int, None], default None
 47        The size of chunks to retrieve from `Pipe.get_data()` for syncing.
 48
 49    kwargs: Any
 50        Additional flags to pass to `Pipe.get_data()` and `Pipe.sync()`, e.g. `workers`.
 51
 52    Returns
 53    -------
 54    A SuccessTuple indicating success.
 55    """
 56    if str(instance_keys) == self.instance_keys:
 57        return False, f"Cannot copy {self} to instance '{instance_keys}'."
 58
 59    begin, end = self.parse_date_bounds(begin, end)
 60
 61    new_pipe = mrsm.Pipe(
 62        self.connector_keys,
 63        self.metric_key,
 64        self.location_key,
 65        parameters=self.parameters.copy(),
 66        instance=instance_keys,
 67    )
 68
 69    new_pipe_is_registered = new_pipe.id is not None
 70
 71    metadata_method = new_pipe.edit if new_pipe_is_registered else new_pipe.register
 72    metadata_success, metadata_msg = metadata_method(debug=debug)
 73    if not metadata_success:
 74        return metadata_success, metadata_msg
 75
 76    if not self.exists(debug=debug):
 77        return True, f"{self} does not exist; nothing to sync."
 78
 79    original_as_iterator = kwargs.get('as_iterator', None)
 80    kwargs['as_iterator'] = True
 81
 82    chunk_generator = self.get_data(
 83        begin=begin,
 84        end=end,
 85        params=params,
 86        chunk_interval=chunk_interval,
 87        debug=debug,
 88        **kwargs
 89    )
 90
 91    if original_as_iterator is None:
 92        _ = kwargs.pop('as_iterator', None)
 93    else:
 94        kwargs['as_iterator'] = original_as_iterator
 95
 96    sync_success, sync_msg = new_pipe.sync(
 97        chunk_generator,
 98        begin=begin,
 99        end=end,
100        params=params,
101        debug=debug,
102        **kwargs
103    )
104    msg = (
105        f"Successfully synced {new_pipe}:\n{sync_msg}"
106        if sync_success
107        else f"Failed to sync {new_pipe}:\n{sync_msg}"
108    )
109    return sync_success, msg

Copy a pipe to another instance.

Parameters
  • instance_keys (str): The instance to which to copy this pipe.
  • sync (bool, default True): If True, sync the source pipe's documents
  • begin (Union[datetime, int, None], default None): Beginning datetime value to pass to Pipe.get_data().
  • end (Union[datetime, int, None], default None): End datetime value to pass to Pipe.get_data().
  • params (Optional[Dict[str, Any]], default None): Parameters filter to pass to Pipe.get_data().
  • chunk_interval (Union[timedelta, int, None], default None): The size of chunks to retrieve from Pipe.get_data() for syncing.
  • kwargs (Any): Additional flags to pass to Pipe.get_data() and Pipe.sync(), e.g. workers.
Returns
  • A SuccessTuple indicating success.
class Plugin:
  50class Plugin:
  51    """Handle packaging of Meerschaum plugins."""
  52
  53    def __init__(
  54        self,
  55        name: str,
  56        version: Optional[str] = None,
  57        user_id: Optional[int] = None,
  58        required: Optional[List[str]] = None,
  59        attributes: Optional[Dict[str, Any]] = None,
  60        archive_path: Optional[pathlib.Path] = None,
  61        venv_path: Optional[pathlib.Path] = None,
  62        repo_connector: Optional['mrsm.connectors.api.APIConnector'] = None,
  63        repo: Union['mrsm.connectors.api.APIConnector', str, None] = None,
  64    ):
  65        import meerschaum.config.paths as paths
  66        from meerschaum._internal.static import STATIC_CONFIG
  67        sep = STATIC_CONFIG['plugins']['repo_separator']
  68        _repo = None
  69        if sep in name:
  70            try:
  71                name, _repo = name.split(sep)
  72            except Exception as e:
  73                error(f"Invalid plugin name: '{name}'")
  74        self._repo_in_name = _repo
  75
  76        if attributes is None:
  77            attributes = {}
  78        self.name = name
  79        self.attributes = attributes
  80        self.user_id = user_id
  81        self._version = version
  82        if required:
  83            self._required = required
  84        self.archive_path = (
  85            archive_path if archive_path is not None
  86            else paths.PLUGINS_ARCHIVES_RESOURCES_PATH / f"{self.name}.tar.gz"
  87        )
  88        self.venv_path = (
  89            venv_path if venv_path is not None
  90            else paths.VIRTENV_RESOURCES_PATH / self.name
  91        )
  92        self._repo_connector = repo_connector
  93        self._repo_keys = repo
  94
  95
  96    @property
  97    def repo_connector(self):
  98        """
  99        Return the repository connector for this plugin.
 100        NOTE: This imports the `connectors` module, which imports certain plugin modules.
 101        """
 102        if self._repo_connector is None:
 103            from meerschaum.connectors.parse import parse_repo_keys
 104
 105            repo_keys = self._repo_keys or self._repo_in_name
 106            if self._repo_in_name and self._repo_keys and self._repo_keys != self._repo_in_name:
 107                error(
 108                    f"Received inconsistent repos: '{self._repo_in_name}' and '{self._repo_keys}'."
 109                )
 110            repo_connector = parse_repo_keys(repo_keys)
 111            self._repo_connector = repo_connector
 112        return self._repo_connector
 113
 114
 115    @property
 116    def version(self):
 117        """
 118        Return the plugin's module version is defined (`__version__`) if it's defined.
 119        """
 120        if self._version is None:
 121            try:
 122                self._version = self.module.__version__
 123            except Exception as e:
 124                self._version = None
 125        return self._version
 126
 127
 128    @property
 129    def module(self):
 130        """
 131        Return the Python module of the underlying plugin,
 132        or `None` if the plugin is not installed or its import failed
 133        (check `Plugin.import_error` for the swallowed exception).
 134        """
 135        if '_module' not in self.__dict__ or self.__dict__.get('_module', None) is None:
 136            from meerschaum.plugins import _get_loaded_module, _get_plugins_package_name
 137            loaded_module = _get_loaded_module(f'{_get_plugins_package_name()}.{self.name}')
 138            if loaded_module is not None:
 139                self._module = loaded_module
 140                return self._module
 141
 142            if self.__file__ is None:
 143                return None
 144
 145            from meerschaum.plugins import import_plugins
 146            self._module = import_plugins(str(self), warn=False)
 147
 148        return self._module
 149
 150    @property
 151    def import_error(self) -> Union[Exception, None]:
 152        """
 153        Return the exception raised by this plugin's most recent failed import
 154        (i.e. when `Plugin.module` is `None` even though the plugin is installed),
 155        otherwise `None`.
 156        """
 157        from meerschaum.plugins import _plugins_import_errors
 158        return _plugins_import_errors.get(self.name, None)
 159
 160
 161    @property
 162    def __file__(self) -> Union[str, None]:
 163        """
 164        Return the file path (str) of the plugin if it exists, otherwise `None`.
 165        """
 166        if self.__dict__.get('_module', None) is not None:
 167            return self.module.__file__
 168
 169        import meerschaum.config.paths as paths
 170
 171        potential_dir = paths.PLUGINS_RESOURCES_PATH / self.name
 172        if (
 173            potential_dir.exists()
 174            and potential_dir.is_dir()
 175            and (potential_dir / '__init__.py').exists()
 176        ):
 177            return str((potential_dir / '__init__.py').as_posix())
 178
 179        potential_file = paths.PLUGINS_RESOURCES_PATH / (self.name + '.py')
 180        if potential_file.exists() and not potential_file.is_dir():
 181            return str(potential_file.as_posix())
 182
 183        return None
 184
 185
 186    @property
 187    def requirements_file_path(self) -> Union[pathlib.Path, None]:
 188        """
 189        If a file named `requirements.txt` exists, return its path.
 190        """
 191        if self.__file__ is None:
 192            return None
 193        path = pathlib.Path(self.__file__).parent / 'requirements.txt'
 194        if not path.exists():
 195            return None
 196        return path
 197
 198
 199    def is_installed(self, **kw) -> bool:
 200        """
 201        Check whether a plugin is correctly installed.
 202
 203        Returns
 204        -------
 205        A `bool` indicating whether a plugin exists and is successfully imported.
 206        """
 207        return self.__file__ is not None
 208
 209
 210    def make_tar(self, debug: bool = False) -> pathlib.Path:
 211        """
 212        Compress the plugin's source files into a `.tar.gz` archive and return the archive's path.
 213
 214        Parameters
 215        ----------
 216        debug: bool, default False
 217            Verbosity toggle.
 218
 219        Returns
 220        -------
 221        A `pathlib.Path` to the archive file's path.
 222
 223        """
 224        import tarfile, pathlib, subprocess, fnmatch
 225        from meerschaum.utils.debug import dprint
 226        from meerschaum.utils.packages import attempt_import
 227        pathspec = attempt_import('pathspec', debug=debug)
 228
 229        if not self.__file__:
 230            from meerschaum.utils.warnings import error
 231            error(f"Could not find file for plugin '{self}'.")
 232        if '__init__.py' in self.__file__ or os.path.isdir(self.__file__):
 233            path = self.__file__.replace('__init__.py', '')
 234            is_dir = True
 235        else:
 236            path = self.__file__
 237            is_dir = False
 238
 239        old_cwd = os.getcwd()
 240        real_parent_path = pathlib.Path(os.path.realpath(path)).parent
 241        os.chdir(real_parent_path)
 242
 243        default_patterns_to_ignore = [
 244            '.pyc',
 245            '__pycache__/',
 246            'eggs/',
 247            '__pypackages__/',
 248            '.git',
 249        ]
 250
 251        def parse_gitignore() -> 'Set[str]':
 252            gitignore_path = pathlib.Path(path) / '.gitignore'
 253            if not gitignore_path.exists():
 254                return set(default_patterns_to_ignore)
 255            with open(gitignore_path, 'r', encoding='utf-8') as f:
 256                gitignore_text = f.read()
 257            return set(pathspec.PathSpec.from_lines(
 258                pathspec.patterns.GitWildMatchPattern,
 259                default_patterns_to_ignore + gitignore_text.splitlines()
 260            ).match_tree(path))
 261
 262        patterns_to_ignore = parse_gitignore() if is_dir else set()
 263
 264        if debug:
 265            dprint(f"Patterns to ignore:\n{patterns_to_ignore}")
 266
 267        with tarfile.open(self.archive_path, 'w:gz') as tarf:
 268            if not is_dir:
 269                tarf.add(f"{self.name}.py")
 270            else:
 271                for root, dirs, files in os.walk(self.name):
 272                    for f in files:
 273                        good_file = True
 274                        fp = os.path.join(root, f)
 275                        for pattern in patterns_to_ignore:
 276                            if pattern in str(fp) or f.startswith('.'):
 277                                good_file = False
 278                                break
 279                        if good_file:
 280                            if debug:
 281                                dprint(f"Adding '{fp}'...")
 282                            tarf.add(fp)
 283
 284        ### clean up and change back to old directory
 285        os.chdir(old_cwd)
 286
 287        ### change to 775 to avoid permissions issues with the API in a Docker container
 288        self.archive_path.chmod(0o775)
 289
 290        if debug:
 291            dprint(f"Created archive '{self.archive_path}'.")
 292        return self.archive_path
 293
 294
 295    def install(
 296        self,
 297        skip_deps: bool = False,
 298        force: bool = False,
 299        debug: bool = False,
 300    ) -> SuccessTuple:
 301        """
 302        Extract a plugin's tar archive to the plugins directory.
 303        
 304        This function checks if the plugin is already installed and if the version is equal or
 305        greater than the existing installation.
 306
 307        Parameters
 308        ----------
 309        skip_deps: bool, default False
 310            If `True`, do not install dependencies.
 311
 312        force: bool, default False
 313            If `True`, continue with installation, even if required packages fail to install.
 314
 315        debug: bool, default False
 316            Verbosity toggle.
 317
 318        Returns
 319        -------
 320        A `SuccessTuple` of success (bool) and a message (str).
 321
 322        """
 323        installation_key = self.full_name
 324        with _ongoing_installations_lock:
 325            if installation_key in _ongoing_installations:
 326                return True, f"Already installing plugin '{self}'."
 327            _ongoing_installations.add(installation_key)
 328
 329        try:
 330            from meerschaum.utils.packages import _pip_install_lock
 331            with _pip_install_lock(self.name):
 332                return self._install_with_rollback(
 333                    skip_deps=skip_deps,
 334                    force=force,
 335                    debug=debug,
 336                )
 337        finally:
 338            with _ongoing_installations_lock:
 339                _ongoing_installations.discard(installation_key)
 340
 341
 342    def _install_with_rollback(
 343        self,
 344        skip_deps: bool = False,
 345        force: bool = False,
 346        debug: bool = False,
 347    ) -> SuccessTuple:
 348        """Restore the previous plugin source and environment after a failed install."""
 349        import tempfile
 350        import meerschaum.config.paths as paths
 351        from meerschaum.plugins import sync_plugins_symlinks
 352        from meerschaum.utils.packages import reload_meerschaum
 353
 354        if not self.archive_path.exists():
 355            return False, f"Missing archive file for plugin '{self}'."
 356
 357        paths.PLUGINS_TEMP_RESOURCES_PATH.mkdir(parents=True, exist_ok=True)
 358        source_paths = [
 359            plugins_dir_path / filename
 360            for plugins_dir_path in paths.PLUGINS_DIR_PATHS
 361            for filename in (self.name, self.name + '.py')
 362        ]
 363        tracked_paths = source_paths + [paths.VIRTENV_RESOURCES_PATH / self.name]
 364
 365        with tempfile.TemporaryDirectory(
 366            prefix=f'.{self.name}-rollback-',
 367            dir=paths.PLUGINS_TEMP_RESOURCES_PATH,
 368        ) as rollback_dir_str:
 369            rollback_dir = pathlib.Path(rollback_dir_str)
 370            backups = {}
 371            rollback_prepared = False
 372
 373            def prepare_rollback() -> None:
 374                nonlocal rollback_prepared
 375                for index, path in enumerate(tracked_paths):
 376                    if not path.exists() and not path.is_symlink():
 377                        continue
 378                    backup_path = rollback_dir / str(index)
 379                    _copy_path(path, backup_path)
 380                    backups[path] = backup_path
 381                rollback_prepared = True
 382
 383            def rollback() -> None:
 384                if not rollback_prepared:
 385                    return
 386                for path in tracked_paths:
 387                    _remove_path(path)
 388                for path, backup_path in backups.items():
 389                    path.parent.mkdir(parents=True, exist_ok=True)
 390                    shutil.move(str(backup_path), str(path))
 391                self.__dict__.pop('_module', None)
 392                self.__dict__.pop('_required', None)
 393                sync_plugins_symlinks(debug=debug)
 394                reload_meerschaum(debug=debug)
 395
 396            try:
 397                result = self._install(
 398                    skip_deps=skip_deps,
 399                    force=force,
 400                    debug=debug,
 401                    _prepare_rollback=prepare_rollback,
 402                )
 403            except BaseException:
 404                rollback()
 405                raise
 406            if not result[0]:
 407                try:
 408                    rollback()
 409                except Exception as e:
 410                    return False, result[1] + f"\nFailed to roll back plugin '{self}': {e}"
 411            return result
 412
 413
 414    def _install(
 415        self,
 416        skip_deps: bool = False,
 417        force: bool = False,
 418        debug: bool = False,
 419        _prepare_rollback: Optional[Any] = None,
 420    ) -> SuccessTuple:
 421
 422        import meerschaum.config.paths as paths
 423        from meerschaum.utils.warnings import warn, error
 424        if debug:
 425            from meerschaum.utils.debug import dprint
 426        import tarfile
 427        import re
 428        import ast
 429        from meerschaum.plugins import sync_plugins_symlinks
 430        from meerschaum.utils.packages import attempt_import, reload_meerschaum
 431        from meerschaum.utils.venv import init_venv
 432        from meerschaum.utils.misc import safely_extract_tar
 433        old_cwd = os.getcwd()
 434        old_version = ''
 435        new_version = ''
 436        temp_dir = paths.PLUGINS_TEMP_RESOURCES_PATH / self.name
 437        temp_dir.mkdir(exist_ok=True)
 438
 439        if not self.archive_path.exists():
 440            return False, f"Missing archive file for plugin '{self}'."
 441        if self.version is not None:
 442            old_version = self.version
 443            if debug:
 444                dprint(f"Found existing version '{old_version}' for plugin '{self}'.")
 445
 446        if debug:
 447            dprint(f"Extracting '{self.archive_path}' to '{temp_dir}'...")
 448
 449        try:
 450            with tarfile.open(self.archive_path, 'r:gz') as tarf:
 451                safely_extract_tar(tarf, temp_dir)
 452        except Exception as e:
 453            warn(e)
 454            return False, f"Failed to extract plugin '{self.name}'."
 455
 456        ### search for version information
 457        files = os.listdir(temp_dir)
 458        
 459        if str(files[0]) == self.name:
 460            is_dir = True
 461        elif str(files[0]) == self.name + '.py':
 462            is_dir = False
 463        else:
 464            error(f"Unknown format encountered for plugin '{self}'.")
 465
 466        fpath = temp_dir / files[0]
 467        if is_dir:
 468            fpath = fpath / '__init__.py'
 469
 470        with open(fpath, 'r', encoding='utf-8') as f:
 471            init_lines = f.readlines()
 472        new_version = None
 473        for line in init_lines:
 474            if '__version__' not in line:
 475                continue
 476            version_match = re.search(r'__version__(\s?)=', line.lstrip().rstrip())
 477            if not version_match:
 478                continue
 479            new_version = ast.literal_eval(line.split('=')[1].lstrip().rstrip())
 480            break
 481        if not new_version:
 482            warn(
 483                f"No `__version__` defined for plugin '{self}'. "
 484                + "Assuming new version...",
 485                stack = False,
 486            )
 487
 488        packaging_version = attempt_import('packaging.version')
 489        try:
 490            is_new_version = (not new_version and not old_version) or (
 491                packaging_version.parse(old_version) < packaging_version.parse(new_version)
 492            )
 493            is_same_version = new_version and old_version and (
 494                packaging_version.parse(old_version) == packaging_version.parse(new_version)
 495            )
 496        except Exception:
 497            is_new_version, is_same_version = True, False
 498
 499        ### Determine where to permanently store the new plugin.
 500        plugin_installation_dir_path = paths.PLUGINS_DIR_PATHS[0]
 501        for path in paths.PLUGINS_DIR_PATHS:
 502            if not path.exists():
 503                warn(f"Plugins path does not exist: {path}", stack=False)
 504                continue
 505
 506            files_in_plugins_dir = os.listdir(path)
 507            if (
 508                self.name in files_in_plugins_dir
 509                or
 510                (self.name + '.py') in files_in_plugins_dir
 511            ):
 512                plugin_installation_dir_path = path
 513                break
 514
 515        success_msg = (
 516            f"Successfully installed plugin '{self}'"
 517            + ("\n    (skipped dependencies)" if skip_deps else "")
 518            + "."
 519        )
 520        success, abort = None, None
 521
 522        if is_same_version and not force:
 523            success, msg = True, (
 524                f"Plugin '{self}' is up-to-date (version {old_version}).\n" +
 525                "    Install again with `-f` or `--force` to reinstall."
 526            )
 527            abort = True
 528        elif is_new_version or force:
 529            if _prepare_rollback is not None:
 530                _prepare_rollback()
 531            source_path = temp_dir / files[0]
 532            destination_path = plugin_installation_dir_path / files[0]
 533            try:
 534                plugin_installation_dir_path.mkdir(parents=True, exist_ok=True)
 535                for existing_path in (
 536                    plugin_installation_dir_path / self.name,
 537                    plugin_installation_dir_path / (self.name + '.py'),
 538                ):
 539                    _remove_path(existing_path)
 540                if debug:
 541                    dprint(f"Moving '{source_path}' to '{destination_path}'...")
 542                shutil.move(str(source_path), str(destination_path))
 543            except Exception as e:
 544                success, msg = False, f"Failed to install plugin '{self}': {e}"
 545            else:
 546                success, msg = True, success_msg
 547        else:
 548            success, msg = False, (
 549                f"Your installed version of plugin '{self}' ({old_version}) is higher than "
 550                + f"attempted version {new_version}."
 551            )
 552
 553        shutil.rmtree(temp_dir)
 554        os.chdir(old_cwd)
 555
 556        ### Reload the plugin's module.
 557        sync_plugins_symlinks(debug=debug)
 558        if '_module' in self.__dict__:
 559            del self.__dict__['_module']
 560        init_venv(venv=self.name, force=True, debug=debug)
 561        reload_meerschaum(debug=debug)
 562
 563        ### Record the origin only after a successful transaction. Locally-installed
 564        ### plugins have no explicit repo and are intentionally skipped.
 565        _origin_repo_keys = (
 566            str(self._repo_connector)
 567            if self._repo_connector is not None
 568            else self._repo_keys or self._repo_in_name
 569        )
 570
 571        def write_origin() -> None:
 572            if not _origin_repo_keys:
 573                return
 574            try:
 575                from meerschaum.plugins._origins import write_plugin_origin
 576                write_plugin_origin(
 577                    self.name,
 578                    _origin_repo_keys,
 579                    plugin_installation_dir_path,
 580                    debug=debug,
 581                )
 582            except Exception:
 583                pass
 584
 585        if abort:
 586            write_origin()
 587
 588        ### if we've already failed, return here
 589        if not success or abort:
 590            return success, msg
 591
 592        ### attempt to install dependencies
 593        dependencies_installed = skip_deps or self.install_dependencies(force=force, debug=debug)
 594        if not dependencies_installed:
 595            return False, f"Failed to install dependencies for plugin '{self}'."
 596
 597        ### handling success tuple, bool, or other (typically None)
 598        setup_tuple = self.setup(debug=debug)
 599        if isinstance(setup_tuple, tuple):
 600            if not setup_tuple[0]:
 601                success, msg = setup_tuple
 602        elif isinstance(setup_tuple, bool):
 603            if not setup_tuple:
 604                success, msg = False, (
 605                    f"Failed to run post-install setup for plugin '{self}'." + '\n' +
 606                    f"Check `setup()` in '{self.__file__}' for more information " +
 607                    "(no error message provided)."
 608                )
 609            else:
 610                success, msg = True, success_msg
 611        elif setup_tuple is None:
 612            success = True
 613            msg = (
 614                f"Post-install for plugin '{self}' returned None. " +
 615                "Assuming plugin successfully installed."
 616            )
 617            warn(msg)
 618        else:
 619            success = False
 620            msg = (
 621                f"Post-install for plugin '{self}' returned unexpected value " +
 622                f"of type '{type(setup_tuple)}': {setup_tuple}"
 623            )
 624
 625        _ = self.module
 626        if success:
 627            write_origin()
 628        return success, msg
 629
 630
 631    def remove_archive(
 632        self,        
 633        debug: bool = False
 634    ) -> SuccessTuple:
 635        """Remove a plugin's archive file."""
 636        if not self.archive_path.exists():
 637            return True, f"Archive file for plugin '{self}' does not exist."
 638        try:
 639            self.archive_path.unlink()
 640        except Exception as e:
 641            return False, f"Failed to remove archive for plugin '{self}':\n{e}"
 642        return True, "Success"
 643
 644
 645    def remove_venv(
 646        self,        
 647        debug: bool = False
 648    ) -> SuccessTuple:
 649        """Remove a plugin's virtual environment."""
 650        if not self.venv_path.exists():
 651            return True, f"Virtual environment for plugin '{self}' does not exist."
 652        try:
 653            shutil.rmtree(self.venv_path)
 654        except Exception as e:
 655            return False, f"Failed to remove virtual environment for plugin '{self}':\n{e}"
 656        return True, "Success"
 657
 658
 659    def uninstall(self, debug: bool = False) -> SuccessTuple:
 660        """
 661        Remove a plugin, its virtual environment, and archive file.
 662        """
 663        from meerschaum.utils.packages import reload_meerschaum
 664        from meerschaum.plugins import sync_plugins_symlinks
 665        from meerschaum.utils.warnings import warn, info
 666        warnings_thrown_count: int = 0
 667        max_warnings: int = 3
 668
 669        if not self.is_installed():
 670            info(
 671                f"Plugin '{self.name}' doesn't seem to be installed.\n    "
 672                + "Checking for artifacts...",
 673                stack = False,
 674            )
 675        else:
 676            real_path = pathlib.Path(os.path.realpath(self.__file__))
 677            try:
 678                if real_path.name == '__init__.py':
 679                    shutil.rmtree(real_path.parent)
 680                else:
 681                    real_path.unlink()
 682            except Exception as e:
 683                warn(f"Could not remove source files for plugin '{self.name}':\n{e}", stack=False)
 684                warnings_thrown_count += 1
 685            else:
 686                info(f"Removed source files for plugin '{self.name}'.")
 687
 688        if self.venv_path.exists():
 689            success, msg = self.remove_venv(debug=debug)
 690            if not success:
 691                warn(msg, stack=False)
 692                warnings_thrown_count += 1
 693            else:
 694                info(f"Removed virtual environment from plugin '{self.name}'.")
 695
 696        success = warnings_thrown_count < max_warnings
 697        try:
 698            from meerschaum.plugins._origins import remove_plugin_origin
 699            remove_plugin_origin(self.name, debug=debug)
 700        except Exception:
 701            pass
 702        sync_plugins_symlinks(debug=debug)
 703        self.deactivate_venv(force=True, debug=debug)
 704        reload_meerschaum(debug=debug)
 705        return success, (
 706            f"Successfully uninstalled plugin '{self}'." if success
 707            else f"Failed to uninstall plugin '{self}'."
 708        )
 709
 710
 711    def setup(self, *args: str, debug: bool = False, **kw: Any) -> Union[SuccessTuple, bool]:
 712        """
 713        If exists, run the plugin's `setup()` function.
 714
 715        Parameters
 716        ----------
 717        *args: str
 718            The positional arguments passed to the `setup()` function.
 719            
 720        debug: bool, default False
 721            Verbosity toggle.
 722
 723        **kw: Any
 724            The keyword arguments passed to the `setup()` function.
 725
 726        Returns
 727        -------
 728        A `SuccessTuple` or `bool` indicating success.
 729
 730        """
 731        from meerschaum.utils.debug import dprint
 732        import inspect
 733        _setup = None
 734        for name, fp in inspect.getmembers(self.module):
 735            if name == 'setup' and inspect.isfunction(fp):
 736                _setup = fp
 737                break
 738
 739        ### assume success if no setup() is found (not necessary)
 740        if _setup is None:
 741            return True
 742
 743        sig = inspect.signature(_setup)
 744        has_debug, has_kw = ('debug' in sig.parameters), False
 745        for k, v in sig.parameters.items():
 746            if '**' in str(v):
 747                has_kw = True
 748                break
 749
 750        _kw = {}
 751        if has_kw:
 752            _kw.update(kw)
 753        if has_debug:
 754            _kw['debug'] = debug
 755
 756        if debug:
 757            dprint(f"Running setup for plugin '{self}'...")
 758        from meerschaum.utils.venv import Venv
 759        try:
 760            with Venv(self, debug=debug):
 761                return_tuple = _setup(*args, **_kw)
 762        except Exception as e:
 763            return False, str(e)
 764
 765        if isinstance(return_tuple, tuple):
 766            return return_tuple
 767        if isinstance(return_tuple, bool):
 768            return return_tuple, f"Setup for Plugin '{self.name}' did not return a message."
 769        if return_tuple is None:
 770            return False, f"Setup for Plugin '{self.name}' returned None."
 771        return False, f"Unknown return value from setup for Plugin '{self.name}': {return_tuple}"
 772
 773
 774    def get_dependencies(
 775        self,
 776        debug: bool = False,
 777    ) -> List[str]:
 778        """
 779        If the Plugin has specified dependencies in a list called `required`, return the list.
 780        
 781        **NOTE:** Dependecies which start with `'plugin:'` are Meerschaum plugins, not pip packages.
 782        Meerschaum plugins may also specify connector keys for a repo after `'@'`.
 783
 784        Parameters
 785        ----------
 786        debug: bool, default False
 787            Verbosity toggle.
 788
 789        Returns
 790        -------
 791        A list of required packages and plugins (str).
 792
 793        """
 794        if '_required' in self.__dict__:
 795            return self._required
 796
 797        ### If the plugin has not yet been imported,
 798        ### infer the dependencies from the source text.
 799        ### This is not super robust, and it doesn't feel right
 800        ### having multiple versions of the logic.
 801        ### This is necessary when determining the activation order
 802        ### without having import the module.
 803        ### For consistency's sake, the module-less method does not cache the requirements.
 804        if self.__dict__.get('_module', None) is None:
 805            file_path = self.__file__
 806            if file_path is None:
 807                return []
 808            with open(file_path, 'r', encoding='utf-8') as f:
 809                text = f.read()
 810
 811            if 'required' not in text:
 812                return []
 813
 814            ### This has some limitations:
 815            ### It relies on `required` being manually declared.
 816            ### We lose the ability to dynamically alter the `required` list,
 817            ### which is why we've kept the module-reliant method below.
 818            import ast, re
 819            ### NOTE: This technically would break 
 820            ### if `required` was the very first line of the file.
 821            req_start_match = re.search(r'\nrequired(:\s*)?.*=', text)
 822            if not req_start_match:
 823                return []
 824            req_start = req_start_match.start()
 825            equals_sign = req_start + text[req_start:].find('=')
 826
 827            ### Dependencies may have brackets within the strings, so push back the index.
 828            first_opening_brace = equals_sign + 1 + text[equals_sign:].find('[')
 829            if first_opening_brace == -1:
 830                return []
 831
 832            next_closing_brace = equals_sign + 1 + text[equals_sign:].find(']')
 833            if next_closing_brace == -1:
 834                return []
 835
 836            start_ix = first_opening_brace + 1
 837            end_ix = next_closing_brace
 838
 839            num_braces = 0
 840            while True:
 841                if '[' not in text[start_ix:end_ix]:
 842                    break
 843                num_braces += 1
 844                start_ix = end_ix
 845                end_ix += text[end_ix + 1:].find(']') + 1
 846
 847            req_end = end_ix + 1
 848            req_text = (
 849                text[(first_opening_brace-1):req_end]
 850                .lstrip()
 851                .replace('=', '', 1)
 852                .lstrip()
 853                .rstrip()
 854            )
 855            try:
 856                required = ast.literal_eval(req_text)
 857            except Exception as e:
 858                warn(
 859                    f"Unable to determine requirements for plugin '{self.name}' "
 860                    + "without importing the module.\n"
 861                    + "    This may be due to dynamically setting the global `required` list.\n"
 862                    + f"    {e}"
 863                )
 864                return []
 865            return required
 866
 867        import inspect
 868        self.activate_venv(dependencies=False, debug=debug)
 869        required = []
 870        for name, val in inspect.getmembers(self.module):
 871            if name == 'required':
 872                required = val
 873                break
 874        self._required = required
 875        self.deactivate_venv(dependencies=False, debug=debug)
 876        return required
 877
 878
 879    def get_required_plugins(self, debug: bool=False) -> List[mrsm.plugins.Plugin]:
 880        """
 881        Return a list of required Plugin objects.
 882        """
 883        from meerschaum.utils.warnings import warn
 884        from meerschaum.config import get_config
 885        from meerschaum._internal.static import STATIC_CONFIG
 886        from meerschaum.connectors.parse import is_valid_connector_keys
 887        plugins = []
 888        _deps = self.get_dependencies(debug=debug)
 889        sep = STATIC_CONFIG['plugins']['repo_separator']
 890        plugin_names = [
 891            _d[len('plugin:'):] for _d in _deps
 892            if _d.startswith('plugin:') and len(_d) > len('plugin:')
 893        ]
 894        default_repo_keys = get_config('meerschaum', 'repository')
 895        skipped_repo_keys = set()
 896
 897        for _plugin_name in plugin_names:
 898            if sep in _plugin_name:
 899                try:
 900                    _plugin_name, _repo_keys = _plugin_name.split(sep)
 901                except Exception:
 902                    _repo_keys = default_repo_keys
 903                    warn(
 904                        f"Invalid repo keys for required plugin '{_plugin_name}'.\n    "
 905                        + f"Will try to use '{_repo_keys}' instead.",
 906                        stack = False,
 907                    )
 908            else:
 909                _repo_keys = default_repo_keys
 910
 911            if _repo_keys in skipped_repo_keys:
 912                continue
 913
 914            if not is_valid_connector_keys(_repo_keys):
 915                warn(
 916                    f"Invalid connector '{_repo_keys}'.\n"
 917                    f"    Skipping required plugins from repository '{_repo_keys}'",
 918                    stack=False,
 919                )
 920                continue
 921
 922            plugins.append(Plugin(_plugin_name, repo=_repo_keys))
 923
 924        return plugins
 925
 926
 927    def get_required_packages(self, debug: bool=False) -> List[str]:
 928        """
 929        Return the required package names (excluding plugins).
 930        """
 931        _deps = self.get_dependencies(debug=debug)
 932        return [_d for _d in _deps if not _d.startswith('plugin:')]
 933
 934
 935    def activate_venv(
 936        self,
 937        dependencies: bool = True,
 938        init_if_not_exists: bool = True,
 939        debug: bool = False,
 940        **kw
 941    ) -> bool:
 942        """
 943        Activate the virtual environments for the plugin and its dependencies.
 944
 945        Parameters
 946        ----------
 947        dependencies: bool, default True
 948            If `True`, activate the virtual environments for required plugins.
 949
 950        Returns
 951        -------
 952        A bool indicating success.
 953        """
 954        import meerschaum.config.paths as paths
 955        from meerschaum.utils.venv import venv_target_path
 956        from meerschaum.utils.packages import activate_venv
 957        from meerschaum.utils.misc import make_symlink, is_symlink
 958
 959        if dependencies:
 960            for plugin in self.get_required_plugins(debug=debug):
 961                plugin.activate_venv(debug=debug, init_if_not_exists=init_if_not_exists, **kw)
 962
 963        vtp = venv_target_path(self.name, debug=debug, allow_nonexistent=True)
 964        venv_meerschaum_path = vtp / 'meerschaum'
 965
 966        try:
 967            success, msg = True, "Success"
 968            if is_symlink(venv_meerschaum_path):
 969                if pathlib.Path(os.path.realpath(venv_meerschaum_path)) != paths.PACKAGE_ROOT_PATH:
 970                    venv_meerschaum_path.unlink()
 971                    success, msg = make_symlink(venv_meerschaum_path, paths.PACKAGE_ROOT_PATH)
 972        except Exception as e:
 973            success, msg = False, str(e)
 974        if not success:
 975            warn(
 976                f"Unable to create symlink {venv_meerschaum_path} to {paths.PACKAGE_ROOT_PATH}:\n"
 977                f"{msg}"
 978            )
 979
 980        return activate_venv(self.name, init_if_not_exists=init_if_not_exists, debug=debug, **kw)
 981
 982
 983    def deactivate_venv(self, dependencies: bool=True, debug: bool = False, **kw) -> bool:
 984        """
 985        Deactivate the virtual environments for the plugin and its dependencies.
 986
 987        Parameters
 988        ----------
 989        dependencies: bool, default True
 990            If `True`, deactivate the virtual environments for required plugins.
 991
 992        Returns
 993        -------
 994        A bool indicating success.
 995        """
 996        from meerschaum.utils.packages import deactivate_venv
 997        success = deactivate_venv(self.name, debug=debug, **kw)
 998        if dependencies:
 999            for plugin in self.get_required_plugins(debug=debug):
1000                plugin.deactivate_venv(debug=debug, **kw)
1001        return success
1002
1003
1004    def install_dependencies(
1005        self,
1006        force: bool = False,
1007        debug: bool = False,
1008    ) -> bool:
1009        """
1010        If specified, install dependencies.
1011        
1012        **NOTE:** Dependencies that start with `'plugin:'` will be installed as
1013        Meerschaum plugins from the same repository as this Plugin.
1014        To install from a different repository, add the repo keys after `'@'`
1015        (e.g. `'plugin:foo@api:bar'`).
1016
1017        Parameters
1018        ----------
1019        force: bool, default False
1020            If `True`, continue with the installation, even if some
1021            required packages fail to install.
1022
1023        debug: bool, default False
1024            Verbosity toggle.
1025
1026        Returns
1027        -------
1028        A bool indicating success.
1029        """
1030        from meerschaum.utils.packages import pip_install, venv_contains_package
1031        from meerschaum.utils.warnings import warn, info
1032        _deps = self.get_dependencies(debug=debug)
1033        if not _deps and self.requirements_file_path is None:
1034            return True
1035
1036        plugins = self.get_required_plugins(debug=debug)
1037        for _plugin in plugins:
1038            if _plugin.name == self.name:
1039                warn(f"Plugin '{self.name}' cannot depend on itself! Skipping...", stack=False)
1040                continue
1041            _success, _msg = _plugin.repo_connector.install_plugin(
1042                _plugin.name, debug=debug, force=force
1043            )
1044            if not _success:
1045                warn(
1046                    f"Failed to install required plugin '{_plugin}' from '{_plugin.repo_connector}'"
1047                    + f" for plugin '{self.name}':\n" + _msg,
1048                    stack = False,
1049                )
1050                if not force:
1051                    warn(
1052                        "Try installing with the `--force` flag to continue anyway.",
1053                        stack = False,
1054                    )
1055                    return False
1056                info(
1057                    "Continuing with installation despite the failure "
1058                    + "(careful, things might be broken!)...",
1059                    icon = False
1060                )
1061
1062
1063        ### First step: parse `requirements.txt` if it exists.
1064        if self.requirements_file_path is not None:
1065            if not pip_install(
1066                requirements_file_path=self.requirements_file_path,
1067                venv=self.name, debug=debug
1068            ):
1069                warn(
1070                    f"Failed to resolve 'requirements.txt' for plugin '{self.name}'.",
1071                    stack = False,
1072                )
1073                if not force:
1074                    warn(
1075                        "Try installing with `--force` to continue anyway.",
1076                        stack = False,
1077                    )
1078                    return False
1079                info(
1080                    "Continuing with installation despite the failure "
1081                    + "(careful, things might be broken!)...",
1082                    icon = False
1083                )
1084
1085
1086        ### Don't reinstall packages that are already included in required plugins.
1087        packages = []
1088        _packages = self.get_required_packages(debug=debug)
1089        accounted_for_packages = set()
1090        for package_name in _packages:
1091            for plugin in plugins:
1092                if venv_contains_package(package_name, plugin.name):
1093                    accounted_for_packages.add(package_name)
1094                    break
1095        packages = [pkg for pkg in _packages if pkg not in accounted_for_packages]
1096
1097        ### Attempt pip packages installation.
1098        if packages:
1099            for package in packages:
1100                if not pip_install(package, venv=self.name, debug=debug):
1101                    warn(
1102                        f"Failed to install required package '{package}'"
1103                        + f" for plugin '{self.name}'.",
1104                        stack = False,
1105                    )
1106                    if not force:
1107                        warn(
1108                            "Try installing with `--force` to continue anyway.",
1109                            stack = False,
1110                        )
1111                        return False
1112                    info(
1113                        "Continuing with installation despite the failure "
1114                        + "(careful, things might be broken!)...",
1115                        icon = False
1116                    )
1117        return True
1118
1119
1120    @property
1121    def full_name(self) -> str:
1122        """
1123        Include the repo keys with the plugin's name.
1124        """
1125        from meerschaum._internal.static import STATIC_CONFIG
1126        sep = STATIC_CONFIG['plugins']['repo_separator']
1127        return self.name + sep + str(self.repo_connector)
1128
1129
1130    def __str__(self):
1131        return self.name
1132
1133
1134    def __repr__(self):
1135        return f"Plugin('{self.name}', repo='{self.repo_connector}')"
1136
1137
1138    def __del__(self):
1139        pass

Handle packaging of Meerschaum plugins.

Plugin( name: str, version: Optional[str] = None, user_id: Optional[int] = None, required: Optional[List[str]] = None, attributes: Optional[Dict[str, Any]] = None, archive_path: Optional[pathlib.Path] = None, venv_path: Optional[pathlib.Path] = None, repo_connector: Optional[meerschaum.connectors.APIConnector] = None, repo: Union[meerschaum.connectors.APIConnector, str, NoneType] = None)
53    def __init__(
54        self,
55        name: str,
56        version: Optional[str] = None,
57        user_id: Optional[int] = None,
58        required: Optional[List[str]] = None,
59        attributes: Optional[Dict[str, Any]] = None,
60        archive_path: Optional[pathlib.Path] = None,
61        venv_path: Optional[pathlib.Path] = None,
62        repo_connector: Optional['mrsm.connectors.api.APIConnector'] = None,
63        repo: Union['mrsm.connectors.api.APIConnector', str, None] = None,
64    ):
65        import meerschaum.config.paths as paths
66        from meerschaum._internal.static import STATIC_CONFIG
67        sep = STATIC_CONFIG['plugins']['repo_separator']
68        _repo = None
69        if sep in name:
70            try:
71                name, _repo = name.split(sep)
72            except Exception as e:
73                error(f"Invalid plugin name: '{name}'")
74        self._repo_in_name = _repo
75
76        if attributes is None:
77            attributes = {}
78        self.name = name
79        self.attributes = attributes
80        self.user_id = user_id
81        self._version = version
82        if required:
83            self._required = required
84        self.archive_path = (
85            archive_path if archive_path is not None
86            else paths.PLUGINS_ARCHIVES_RESOURCES_PATH / f"{self.name}.tar.gz"
87        )
88        self.venv_path = (
89            venv_path if venv_path is not None
90            else paths.VIRTENV_RESOURCES_PATH / self.name
91        )
92        self._repo_connector = repo_connector
93        self._repo_keys = repo
name
attributes
user_id
archive_path
venv_path
repo_connector
 96    @property
 97    def repo_connector(self):
 98        """
 99        Return the repository connector for this plugin.
100        NOTE: This imports the `connectors` module, which imports certain plugin modules.
101        """
102        if self._repo_connector is None:
103            from meerschaum.connectors.parse import parse_repo_keys
104
105            repo_keys = self._repo_keys or self._repo_in_name
106            if self._repo_in_name and self._repo_keys and self._repo_keys != self._repo_in_name:
107                error(
108                    f"Received inconsistent repos: '{self._repo_in_name}' and '{self._repo_keys}'."
109                )
110            repo_connector = parse_repo_keys(repo_keys)
111            self._repo_connector = repo_connector
112        return self._repo_connector

Return the repository connector for this plugin. NOTE: This imports the connectors module, which imports certain plugin modules.

version
115    @property
116    def version(self):
117        """
118        Return the plugin's module version is defined (`__version__`) if it's defined.
119        """
120        if self._version is None:
121            try:
122                self._version = self.module.__version__
123            except Exception as e:
124                self._version = None
125        return self._version

Return the plugin's module version is defined (__version__) if it's defined.

module
128    @property
129    def module(self):
130        """
131        Return the Python module of the underlying plugin,
132        or `None` if the plugin is not installed or its import failed
133        (check `Plugin.import_error` for the swallowed exception).
134        """
135        if '_module' not in self.__dict__ or self.__dict__.get('_module', None) is None:
136            from meerschaum.plugins import _get_loaded_module, _get_plugins_package_name
137            loaded_module = _get_loaded_module(f'{_get_plugins_package_name()}.{self.name}')
138            if loaded_module is not None:
139                self._module = loaded_module
140                return self._module
141
142            if self.__file__ is None:
143                return None
144
145            from meerschaum.plugins import import_plugins
146            self._module = import_plugins(str(self), warn=False)
147
148        return self._module

Return the Python module of the underlying plugin, or None if the plugin is not installed or its import failed (check Plugin.import_error for the swallowed exception).

import_error: Optional[Exception]
150    @property
151    def import_error(self) -> Union[Exception, None]:
152        """
153        Return the exception raised by this plugin's most recent failed import
154        (i.e. when `Plugin.module` is `None` even though the plugin is installed),
155        otherwise `None`.
156        """
157        from meerschaum.plugins import _plugins_import_errors
158        return _plugins_import_errors.get(self.name, None)

Return the exception raised by this plugin's most recent failed import (i.e. when Plugin.module is None even though the plugin is installed), otherwise None.

requirements_file_path: Optional[pathlib.Path]
186    @property
187    def requirements_file_path(self) -> Union[pathlib.Path, None]:
188        """
189        If a file named `requirements.txt` exists, return its path.
190        """
191        if self.__file__ is None:
192            return None
193        path = pathlib.Path(self.__file__).parent / 'requirements.txt'
194        if not path.exists():
195            return None
196        return path

If a file named requirements.txt exists, return its path.

def is_installed(self, **kw) -> bool:
199    def is_installed(self, **kw) -> bool:
200        """
201        Check whether a plugin is correctly installed.
202
203        Returns
204        -------
205        A `bool` indicating whether a plugin exists and is successfully imported.
206        """
207        return self.__file__ is not None

Check whether a plugin is correctly installed.

Returns
  • A bool indicating whether a plugin exists and is successfully imported.
def make_tar(self, debug: bool = False) -> pathlib.Path:
210    def make_tar(self, debug: bool = False) -> pathlib.Path:
211        """
212        Compress the plugin's source files into a `.tar.gz` archive and return the archive's path.
213
214        Parameters
215        ----------
216        debug: bool, default False
217            Verbosity toggle.
218
219        Returns
220        -------
221        A `pathlib.Path` to the archive file's path.
222
223        """
224        import tarfile, pathlib, subprocess, fnmatch
225        from meerschaum.utils.debug import dprint
226        from meerschaum.utils.packages import attempt_import
227        pathspec = attempt_import('pathspec', debug=debug)
228
229        if not self.__file__:
230            from meerschaum.utils.warnings import error
231            error(f"Could not find file for plugin '{self}'.")
232        if '__init__.py' in self.__file__ or os.path.isdir(self.__file__):
233            path = self.__file__.replace('__init__.py', '')
234            is_dir = True
235        else:
236            path = self.__file__
237            is_dir = False
238
239        old_cwd = os.getcwd()
240        real_parent_path = pathlib.Path(os.path.realpath(path)).parent
241        os.chdir(real_parent_path)
242
243        default_patterns_to_ignore = [
244            '.pyc',
245            '__pycache__/',
246            'eggs/',
247            '__pypackages__/',
248            '.git',
249        ]
250
251        def parse_gitignore() -> 'Set[str]':
252            gitignore_path = pathlib.Path(path) / '.gitignore'
253            if not gitignore_path.exists():
254                return set(default_patterns_to_ignore)
255            with open(gitignore_path, 'r', encoding='utf-8') as f:
256                gitignore_text = f.read()
257            return set(pathspec.PathSpec.from_lines(
258                pathspec.patterns.GitWildMatchPattern,
259                default_patterns_to_ignore + gitignore_text.splitlines()
260            ).match_tree(path))
261
262        patterns_to_ignore = parse_gitignore() if is_dir else set()
263
264        if debug:
265            dprint(f"Patterns to ignore:\n{patterns_to_ignore}")
266
267        with tarfile.open(self.archive_path, 'w:gz') as tarf:
268            if not is_dir:
269                tarf.add(f"{self.name}.py")
270            else:
271                for root, dirs, files in os.walk(self.name):
272                    for f in files:
273                        good_file = True
274                        fp = os.path.join(root, f)
275                        for pattern in patterns_to_ignore:
276                            if pattern in str(fp) or f.startswith('.'):
277                                good_file = False
278                                break
279                        if good_file:
280                            if debug:
281                                dprint(f"Adding '{fp}'...")
282                            tarf.add(fp)
283
284        ### clean up and change back to old directory
285        os.chdir(old_cwd)
286
287        ### change to 775 to avoid permissions issues with the API in a Docker container
288        self.archive_path.chmod(0o775)
289
290        if debug:
291            dprint(f"Created archive '{self.archive_path}'.")
292        return self.archive_path

Compress the plugin's source files into a .tar.gz archive and return the archive's path.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
  • A pathlib.Path to the archive file's path.
def install( self, skip_deps: bool = False, force: bool = False, debug: bool = False) -> Tuple[bool, str]:
295    def install(
296        self,
297        skip_deps: bool = False,
298        force: bool = False,
299        debug: bool = False,
300    ) -> SuccessTuple:
301        """
302        Extract a plugin's tar archive to the plugins directory.
303        
304        This function checks if the plugin is already installed and if the version is equal or
305        greater than the existing installation.
306
307        Parameters
308        ----------
309        skip_deps: bool, default False
310            If `True`, do not install dependencies.
311
312        force: bool, default False
313            If `True`, continue with installation, even if required packages fail to install.
314
315        debug: bool, default False
316            Verbosity toggle.
317
318        Returns
319        -------
320        A `SuccessTuple` of success (bool) and a message (str).
321
322        """
323        installation_key = self.full_name
324        with _ongoing_installations_lock:
325            if installation_key in _ongoing_installations:
326                return True, f"Already installing plugin '{self}'."
327            _ongoing_installations.add(installation_key)
328
329        try:
330            from meerschaum.utils.packages import _pip_install_lock
331            with _pip_install_lock(self.name):
332                return self._install_with_rollback(
333                    skip_deps=skip_deps,
334                    force=force,
335                    debug=debug,
336                )
337        finally:
338            with _ongoing_installations_lock:
339                _ongoing_installations.discard(installation_key)

Extract a plugin's tar archive to the plugins directory.

This function checks if the plugin is already installed and if the version is equal or greater than the existing installation.

Parameters
  • skip_deps (bool, default False): If True, do not install dependencies.
  • force (bool, default False): If True, continue with installation, even if required packages fail to install.
  • debug (bool, default False): Verbosity toggle.
Returns
def remove_archive(self, debug: bool = False) -> Tuple[bool, str]:
631    def remove_archive(
632        self,        
633        debug: bool = False
634    ) -> SuccessTuple:
635        """Remove a plugin's archive file."""
636        if not self.archive_path.exists():
637            return True, f"Archive file for plugin '{self}' does not exist."
638        try:
639            self.archive_path.unlink()
640        except Exception as e:
641            return False, f"Failed to remove archive for plugin '{self}':\n{e}"
642        return True, "Success"

Remove a plugin's archive file.

def remove_venv(self, debug: bool = False) -> Tuple[bool, str]:
645    def remove_venv(
646        self,        
647        debug: bool = False
648    ) -> SuccessTuple:
649        """Remove a plugin's virtual environment."""
650        if not self.venv_path.exists():
651            return True, f"Virtual environment for plugin '{self}' does not exist."
652        try:
653            shutil.rmtree(self.venv_path)
654        except Exception as e:
655            return False, f"Failed to remove virtual environment for plugin '{self}':\n{e}"
656        return True, "Success"

Remove a plugin's virtual environment.

def uninstall(self, debug: bool = False) -> Tuple[bool, str]:
659    def uninstall(self, debug: bool = False) -> SuccessTuple:
660        """
661        Remove a plugin, its virtual environment, and archive file.
662        """
663        from meerschaum.utils.packages import reload_meerschaum
664        from meerschaum.plugins import sync_plugins_symlinks
665        from meerschaum.utils.warnings import warn, info
666        warnings_thrown_count: int = 0
667        max_warnings: int = 3
668
669        if not self.is_installed():
670            info(
671                f"Plugin '{self.name}' doesn't seem to be installed.\n    "
672                + "Checking for artifacts...",
673                stack = False,
674            )
675        else:
676            real_path = pathlib.Path(os.path.realpath(self.__file__))
677            try:
678                if real_path.name == '__init__.py':
679                    shutil.rmtree(real_path.parent)
680                else:
681                    real_path.unlink()
682            except Exception as e:
683                warn(f"Could not remove source files for plugin '{self.name}':\n{e}", stack=False)
684                warnings_thrown_count += 1
685            else:
686                info(f"Removed source files for plugin '{self.name}'.")
687
688        if self.venv_path.exists():
689            success, msg = self.remove_venv(debug=debug)
690            if not success:
691                warn(msg, stack=False)
692                warnings_thrown_count += 1
693            else:
694                info(f"Removed virtual environment from plugin '{self.name}'.")
695
696        success = warnings_thrown_count < max_warnings
697        try:
698            from meerschaum.plugins._origins import remove_plugin_origin
699            remove_plugin_origin(self.name, debug=debug)
700        except Exception:
701            pass
702        sync_plugins_symlinks(debug=debug)
703        self.deactivate_venv(force=True, debug=debug)
704        reload_meerschaum(debug=debug)
705        return success, (
706            f"Successfully uninstalled plugin '{self}'." if success
707            else f"Failed to uninstall plugin '{self}'."
708        )

Remove a plugin, its virtual environment, and archive file.

def setup( self, *args: str, debug: bool = False, **kw: Any) -> Union[Tuple[bool, str], bool]:
711    def setup(self, *args: str, debug: bool = False, **kw: Any) -> Union[SuccessTuple, bool]:
712        """
713        If exists, run the plugin's `setup()` function.
714
715        Parameters
716        ----------
717        *args: str
718            The positional arguments passed to the `setup()` function.
719            
720        debug: bool, default False
721            Verbosity toggle.
722
723        **kw: Any
724            The keyword arguments passed to the `setup()` function.
725
726        Returns
727        -------
728        A `SuccessTuple` or `bool` indicating success.
729
730        """
731        from meerschaum.utils.debug import dprint
732        import inspect
733        _setup = None
734        for name, fp in inspect.getmembers(self.module):
735            if name == 'setup' and inspect.isfunction(fp):
736                _setup = fp
737                break
738
739        ### assume success if no setup() is found (not necessary)
740        if _setup is None:
741            return True
742
743        sig = inspect.signature(_setup)
744        has_debug, has_kw = ('debug' in sig.parameters), False
745        for k, v in sig.parameters.items():
746            if '**' in str(v):
747                has_kw = True
748                break
749
750        _kw = {}
751        if has_kw:
752            _kw.update(kw)
753        if has_debug:
754            _kw['debug'] = debug
755
756        if debug:
757            dprint(f"Running setup for plugin '{self}'...")
758        from meerschaum.utils.venv import Venv
759        try:
760            with Venv(self, debug=debug):
761                return_tuple = _setup(*args, **_kw)
762        except Exception as e:
763            return False, str(e)
764
765        if isinstance(return_tuple, tuple):
766            return return_tuple
767        if isinstance(return_tuple, bool):
768            return return_tuple, f"Setup for Plugin '{self.name}' did not return a message."
769        if return_tuple is None:
770            return False, f"Setup for Plugin '{self.name}' returned None."
771        return False, f"Unknown return value from setup for Plugin '{self.name}': {return_tuple}"

If exists, run the plugin's setup() function.

Parameters
  • *args (str): The positional arguments passed to the setup() function.
  • debug (bool, default False): Verbosity toggle.
  • **kw (Any): The keyword arguments passed to the setup() function.
Returns
def get_dependencies(self, debug: bool = False) -> List[str]:
774    def get_dependencies(
775        self,
776        debug: bool = False,
777    ) -> List[str]:
778        """
779        If the Plugin has specified dependencies in a list called `required`, return the list.
780        
781        **NOTE:** Dependecies which start with `'plugin:'` are Meerschaum plugins, not pip packages.
782        Meerschaum plugins may also specify connector keys for a repo after `'@'`.
783
784        Parameters
785        ----------
786        debug: bool, default False
787            Verbosity toggle.
788
789        Returns
790        -------
791        A list of required packages and plugins (str).
792
793        """
794        if '_required' in self.__dict__:
795            return self._required
796
797        ### If the plugin has not yet been imported,
798        ### infer the dependencies from the source text.
799        ### This is not super robust, and it doesn't feel right
800        ### having multiple versions of the logic.
801        ### This is necessary when determining the activation order
802        ### without having import the module.
803        ### For consistency's sake, the module-less method does not cache the requirements.
804        if self.__dict__.get('_module', None) is None:
805            file_path = self.__file__
806            if file_path is None:
807                return []
808            with open(file_path, 'r', encoding='utf-8') as f:
809                text = f.read()
810
811            if 'required' not in text:
812                return []
813
814            ### This has some limitations:
815            ### It relies on `required` being manually declared.
816            ### We lose the ability to dynamically alter the `required` list,
817            ### which is why we've kept the module-reliant method below.
818            import ast, re
819            ### NOTE: This technically would break 
820            ### if `required` was the very first line of the file.
821            req_start_match = re.search(r'\nrequired(:\s*)?.*=', text)
822            if not req_start_match:
823                return []
824            req_start = req_start_match.start()
825            equals_sign = req_start + text[req_start:].find('=')
826
827            ### Dependencies may have brackets within the strings, so push back the index.
828            first_opening_brace = equals_sign + 1 + text[equals_sign:].find('[')
829            if first_opening_brace == -1:
830                return []
831
832            next_closing_brace = equals_sign + 1 + text[equals_sign:].find(']')
833            if next_closing_brace == -1:
834                return []
835
836            start_ix = first_opening_brace + 1
837            end_ix = next_closing_brace
838
839            num_braces = 0
840            while True:
841                if '[' not in text[start_ix:end_ix]:
842                    break
843                num_braces += 1
844                start_ix = end_ix
845                end_ix += text[end_ix + 1:].find(']') + 1
846
847            req_end = end_ix + 1
848            req_text = (
849                text[(first_opening_brace-1):req_end]
850                .lstrip()
851                .replace('=', '', 1)
852                .lstrip()
853                .rstrip()
854            )
855            try:
856                required = ast.literal_eval(req_text)
857            except Exception as e:
858                warn(
859                    f"Unable to determine requirements for plugin '{self.name}' "
860                    + "without importing the module.\n"
861                    + "    This may be due to dynamically setting the global `required` list.\n"
862                    + f"    {e}"
863                )
864                return []
865            return required
866
867        import inspect
868        self.activate_venv(dependencies=False, debug=debug)
869        required = []
870        for name, val in inspect.getmembers(self.module):
871            if name == 'required':
872                required = val
873                break
874        self._required = required
875        self.deactivate_venv(dependencies=False, debug=debug)
876        return required

If the Plugin has specified dependencies in a list called required, return the list.

NOTE: Dependecies which start with 'plugin:' are Meerschaum plugins, not pip packages. Meerschaum plugins may also specify connector keys for a repo after '@'.

Parameters
  • debug (bool, default False): Verbosity toggle.
Returns
  • A list of required packages and plugins (str).
def get_required_plugins(self, debug: bool = False) -> List[Plugin]:
879    def get_required_plugins(self, debug: bool=False) -> List[mrsm.plugins.Plugin]:
880        """
881        Return a list of required Plugin objects.
882        """
883        from meerschaum.utils.warnings import warn
884        from meerschaum.config import get_config
885        from meerschaum._internal.static import STATIC_CONFIG
886        from meerschaum.connectors.parse import is_valid_connector_keys
887        plugins = []
888        _deps = self.get_dependencies(debug=debug)
889        sep = STATIC_CONFIG['plugins']['repo_separator']
890        plugin_names = [
891            _d[len('plugin:'):] for _d in _deps
892            if _d.startswith('plugin:') and len(_d) > len('plugin:')
893        ]
894        default_repo_keys = get_config('meerschaum', 'repository')
895        skipped_repo_keys = set()
896
897        for _plugin_name in plugin_names:
898            if sep in _plugin_name:
899                try:
900                    _plugin_name, _repo_keys = _plugin_name.split(sep)
901                except Exception:
902                    _repo_keys = default_repo_keys
903                    warn(
904                        f"Invalid repo keys for required plugin '{_plugin_name}'.\n    "
905                        + f"Will try to use '{_repo_keys}' instead.",
906                        stack = False,
907                    )
908            else:
909                _repo_keys = default_repo_keys
910
911            if _repo_keys in skipped_repo_keys:
912                continue
913
914            if not is_valid_connector_keys(_repo_keys):
915                warn(
916                    f"Invalid connector '{_repo_keys}'.\n"
917                    f"    Skipping required plugins from repository '{_repo_keys}'",
918                    stack=False,
919                )
920                continue
921
922            plugins.append(Plugin(_plugin_name, repo=_repo_keys))
923
924        return plugins

Return a list of required Plugin objects.

def get_required_packages(self, debug: bool = False) -> List[str]:
927    def get_required_packages(self, debug: bool=False) -> List[str]:
928        """
929        Return the required package names (excluding plugins).
930        """
931        _deps = self.get_dependencies(debug=debug)
932        return [_d for _d in _deps if not _d.startswith('plugin:')]

Return the required package names (excluding plugins).

def activate_venv( self, dependencies: bool = True, init_if_not_exists: bool = True, debug: bool = False, **kw) -> bool:
935    def activate_venv(
936        self,
937        dependencies: bool = True,
938        init_if_not_exists: bool = True,
939        debug: bool = False,
940        **kw
941    ) -> bool:
942        """
943        Activate the virtual environments for the plugin and its dependencies.
944
945        Parameters
946        ----------
947        dependencies: bool, default True
948            If `True`, activate the virtual environments for required plugins.
949
950        Returns
951        -------
952        A bool indicating success.
953        """
954        import meerschaum.config.paths as paths
955        from meerschaum.utils.venv import venv_target_path
956        from meerschaum.utils.packages import activate_venv
957        from meerschaum.utils.misc import make_symlink, is_symlink
958
959        if dependencies:
960            for plugin in self.get_required_plugins(debug=debug):
961                plugin.activate_venv(debug=debug, init_if_not_exists=init_if_not_exists, **kw)
962
963        vtp = venv_target_path(self.name, debug=debug, allow_nonexistent=True)
964        venv_meerschaum_path = vtp / 'meerschaum'
965
966        try:
967            success, msg = True, "Success"
968            if is_symlink(venv_meerschaum_path):
969                if pathlib.Path(os.path.realpath(venv_meerschaum_path)) != paths.PACKAGE_ROOT_PATH:
970                    venv_meerschaum_path.unlink()
971                    success, msg = make_symlink(venv_meerschaum_path, paths.PACKAGE_ROOT_PATH)
972        except Exception as e:
973            success, msg = False, str(e)
974        if not success:
975            warn(
976                f"Unable to create symlink {venv_meerschaum_path} to {paths.PACKAGE_ROOT_PATH}:\n"
977                f"{msg}"
978            )
979
980        return activate_venv(self.name, init_if_not_exists=init_if_not_exists, debug=debug, **kw)

Activate the virtual environments for the plugin and its dependencies.

Parameters
  • dependencies (bool, default True): If True, activate the virtual environments for required plugins.
Returns
  • A bool indicating success.
def deactivate_venv(self, dependencies: bool = True, debug: bool = False, **kw) -> bool:
 983    def deactivate_venv(self, dependencies: bool=True, debug: bool = False, **kw) -> bool:
 984        """
 985        Deactivate the virtual environments for the plugin and its dependencies.
 986
 987        Parameters
 988        ----------
 989        dependencies: bool, default True
 990            If `True`, deactivate the virtual environments for required plugins.
 991
 992        Returns
 993        -------
 994        A bool indicating success.
 995        """
 996        from meerschaum.utils.packages import deactivate_venv
 997        success = deactivate_venv(self.name, debug=debug, **kw)
 998        if dependencies:
 999            for plugin in self.get_required_plugins(debug=debug):
1000                plugin.deactivate_venv(debug=debug, **kw)
1001        return success

Deactivate the virtual environments for the plugin and its dependencies.

Parameters
  • dependencies (bool, default True): If True, deactivate the virtual environments for required plugins.
Returns
  • A bool indicating success.
def install_dependencies(self, force: bool = False, debug: bool = False) -> bool:
1004    def install_dependencies(
1005        self,
1006        force: bool = False,
1007        debug: bool = False,
1008    ) -> bool:
1009        """
1010        If specified, install dependencies.
1011        
1012        **NOTE:** Dependencies that start with `'plugin:'` will be installed as
1013        Meerschaum plugins from the same repository as this Plugin.
1014        To install from a different repository, add the repo keys after `'@'`
1015        (e.g. `'plugin:foo@api:bar'`).
1016
1017        Parameters
1018        ----------
1019        force: bool, default False
1020            If `True`, continue with the installation, even if some
1021            required packages fail to install.
1022
1023        debug: bool, default False
1024            Verbosity toggle.
1025
1026        Returns
1027        -------
1028        A bool indicating success.
1029        """
1030        from meerschaum.utils.packages import pip_install, venv_contains_package
1031        from meerschaum.utils.warnings import warn, info
1032        _deps = self.get_dependencies(debug=debug)
1033        if not _deps and self.requirements_file_path is None:
1034            return True
1035
1036        plugins = self.get_required_plugins(debug=debug)
1037        for _plugin in plugins:
1038            if _plugin.name == self.name:
1039                warn(f"Plugin '{self.name}' cannot depend on itself! Skipping...", stack=False)
1040                continue
1041            _success, _msg = _plugin.repo_connector.install_plugin(
1042                _plugin.name, debug=debug, force=force
1043            )
1044            if not _success:
1045                warn(
1046                    f"Failed to install required plugin '{_plugin}' from '{_plugin.repo_connector}'"
1047                    + f" for plugin '{self.name}':\n" + _msg,
1048                    stack = False,
1049                )
1050                if not force:
1051                    warn(
1052                        "Try installing with the `--force` flag to continue anyway.",
1053                        stack = False,
1054                    )
1055                    return False
1056                info(
1057                    "Continuing with installation despite the failure "
1058                    + "(careful, things might be broken!)...",
1059                    icon = False
1060                )
1061
1062
1063        ### First step: parse `requirements.txt` if it exists.
1064        if self.requirements_file_path is not None:
1065            if not pip_install(
1066                requirements_file_path=self.requirements_file_path,
1067                venv=self.name, debug=debug
1068            ):
1069                warn(
1070                    f"Failed to resolve 'requirements.txt' for plugin '{self.name}'.",
1071                    stack = False,
1072                )
1073                if not force:
1074                    warn(
1075                        "Try installing with `--force` to continue anyway.",
1076                        stack = False,
1077                    )
1078                    return False
1079                info(
1080                    "Continuing with installation despite the failure "
1081                    + "(careful, things might be broken!)...",
1082                    icon = False
1083                )
1084
1085
1086        ### Don't reinstall packages that are already included in required plugins.
1087        packages = []
1088        _packages = self.get_required_packages(debug=debug)
1089        accounted_for_packages = set()
1090        for package_name in _packages:
1091            for plugin in plugins:
1092                if venv_contains_package(package_name, plugin.name):
1093                    accounted_for_packages.add(package_name)
1094                    break
1095        packages = [pkg for pkg in _packages if pkg not in accounted_for_packages]
1096
1097        ### Attempt pip packages installation.
1098        if packages:
1099            for package in packages:
1100                if not pip_install(package, venv=self.name, debug=debug):
1101                    warn(
1102                        f"Failed to install required package '{package}'"
1103                        + f" for plugin '{self.name}'.",
1104                        stack = False,
1105                    )
1106                    if not force:
1107                        warn(
1108                            "Try installing with `--force` to continue anyway.",
1109                            stack = False,
1110                        )
1111                        return False
1112                    info(
1113                        "Continuing with installation despite the failure "
1114                        + "(careful, things might be broken!)...",
1115                        icon = False
1116                    )
1117        return True

If specified, install dependencies.

NOTE: Dependencies that start with 'plugin:' will be installed as Meerschaum plugins from the same repository as this Plugin. To install from a different repository, add the repo keys after '@' (e.g. 'plugin:foo@api:bar').

Parameters
  • force (bool, default False): If True, continue with the installation, even if some required packages fail to install.
  • debug (bool, default False): Verbosity toggle.
Returns
  • A bool indicating success.
full_name: str
1120    @property
1121    def full_name(self) -> str:
1122        """
1123        Include the repo keys with the plugin's name.
1124        """
1125        from meerschaum._internal.static import STATIC_CONFIG
1126        sep = STATIC_CONFIG['plugins']['repo_separator']
1127        return self.name + sep + str(self.repo_connector)

Include the repo keys with the plugin's name.

SuccessTuple = typing.Tuple[bool, str]
class Venv:
 19class Venv:
 20    """
 21    Manage a virtual enviroment's activation status.
 22
 23    Examples
 24    --------
 25    >>> from meerschaum.plugins import Plugin
 26    >>> with Venv('mrsm') as venv:
 27    ...     import pandas
 28    >>> with Venv(Plugin('noaa')) as venv:
 29    ...     import requests
 30    >>> venv = Venv('mrsm')
 31    >>> venv.activate()
 32    True
 33    >>> venv.deactivate()
 34    True
 35    >>> 
 36    """
 37
 38    def __init__(
 39        self,
 40        venv: Union[str, 'mrsm.core.Plugin', None] = 'mrsm',
 41        init_if_not_exists: bool = True,
 42        debug: bool = False,
 43    ) -> None:
 44        from meerschaum.utils.venv import activate_venv, deactivate_venv, active_venvs
 45        ### For some weird threading issue,
 46        ### we can't use `isinstance` here.
 47        if '_Plugin' in str(type(venv)):
 48            self._venv = venv.name
 49            self._activate = venv.activate_venv
 50            self._deactivate = venv.deactivate_venv
 51            self._kwargs = {}
 52        else:
 53            self._venv = venv
 54            self._activate = activate_venv
 55            self._deactivate = deactivate_venv
 56            self._kwargs = {'venv': venv}
 57        self._debug = debug
 58        self._init_if_not_exists = init_if_not_exists
 59        ### In case someone calls `deactivate()` before `activate()`.
 60        self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs)
 61
 62
 63    def activate(self, debug: bool = False) -> bool:
 64        """
 65        Activate this virtual environment.
 66        If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments
 67        will also be activated.
 68        """
 69        from meerschaum.utils.venv import active_venvs, init_venv
 70        self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs)
 71        try:
 72            return self._activate(
 73                debug=(debug or self._debug),
 74                init_if_not_exists=self._init_if_not_exists,
 75                **self._kwargs
 76            )
 77        except OSError as e:
 78            if self._init_if_not_exists:
 79                if not init_venv(self._venv, force=True):
 80                    raise e
 81        return self._activate(
 82            debug=(debug or self._debug),
 83            init_if_not_exists=self._init_if_not_exists,
 84            **self._kwargs
 85        )
 86
 87
 88    def deactivate(self, debug: bool = False) -> bool:
 89        """
 90        Deactivate this virtual environment.
 91        If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments
 92        will also be deactivated.
 93        """
 94        return self._deactivate(debug=(debug or self._debug), **self._kwargs)
 95
 96
 97    @property
 98    def target_path(self) -> pathlib.Path:
 99        """
100        Return the target site-packages path for this virtual environment.
101        A `meerschaum.utils.venv.Venv` may have one virtual environment per minor Python version
102        (e.g. Python 3.10 and Python 3.7).
103        """
104        from meerschaum.utils.venv import venv_target_path
105        return venv_target_path(venv=self._venv, allow_nonexistent=True, debug=self._debug)
106
107
108    @property
109    def root_path(self) -> pathlib.Path:
110        """
111        Return the top-level path for this virtual environment.
112        """
113        import meerschaum.config.paths as paths
114        if self._venv is None:
115            return self.target_path.parent
116        return paths.VIRTENV_RESOURCES_PATH / self._venv
117
118
119    def __enter__(self) -> None:
120        self.activate(debug=self._debug)
121
122
123    def __exit__(self, exc_type, exc_value, exc_traceback) -> None:
124        self.deactivate(debug=self._debug)
125
126
127    def __str__(self) -> str:
128        quote = "'" if self._venv is not None else ""
129        return "Venv(" + quote + str(self._venv) + quote + ")"
130
131
132    def __repr__(self) -> str:
133        return self.__str__()

Manage a virtual enviroment's activation status.

Examples
>>> from meerschaum.plugins import Plugin
>>> with Venv('mrsm') as venv:
...     import pandas
>>> with Venv(Plugin('noaa')) as venv:
...     import requests
>>> venv = Venv('mrsm')
>>> venv.activate()
True
>>> venv.deactivate()
True
>>>
Venv( venv: Union[str, Plugin, NoneType] = 'mrsm', init_if_not_exists: bool = True, debug: bool = False)
38    def __init__(
39        self,
40        venv: Union[str, 'mrsm.core.Plugin', None] = 'mrsm',
41        init_if_not_exists: bool = True,
42        debug: bool = False,
43    ) -> None:
44        from meerschaum.utils.venv import activate_venv, deactivate_venv, active_venvs
45        ### For some weird threading issue,
46        ### we can't use `isinstance` here.
47        if '_Plugin' in str(type(venv)):
48            self._venv = venv.name
49            self._activate = venv.activate_venv
50            self._deactivate = venv.deactivate_venv
51            self._kwargs = {}
52        else:
53            self._venv = venv
54            self._activate = activate_venv
55            self._deactivate = deactivate_venv
56            self._kwargs = {'venv': venv}
57        self._debug = debug
58        self._init_if_not_exists = init_if_not_exists
59        ### In case someone calls `deactivate()` before `activate()`.
60        self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs)
def activate(self, debug: bool = False) -> bool:
63    def activate(self, debug: bool = False) -> bool:
64        """
65        Activate this virtual environment.
66        If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments
67        will also be activated.
68        """
69        from meerschaum.utils.venv import active_venvs, init_venv
70        self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs)
71        try:
72            return self._activate(
73                debug=(debug or self._debug),
74                init_if_not_exists=self._init_if_not_exists,
75                **self._kwargs
76            )
77        except OSError as e:
78            if self._init_if_not_exists:
79                if not init_venv(self._venv, force=True):
80                    raise e
81        return self._activate(
82            debug=(debug or self._debug),
83            init_if_not_exists=self._init_if_not_exists,
84            **self._kwargs
85        )

Activate this virtual environment. If a meerschaum.plugins.Plugin was provided, its dependent virtual environments will also be activated.

def deactivate(self, debug: bool = False) -> bool:
88    def deactivate(self, debug: bool = False) -> bool:
89        """
90        Deactivate this virtual environment.
91        If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments
92        will also be deactivated.
93        """
94        return self._deactivate(debug=(debug or self._debug), **self._kwargs)

Deactivate this virtual environment. If a meerschaum.plugins.Plugin was provided, its dependent virtual environments will also be deactivated.

target_path: pathlib.Path
 97    @property
 98    def target_path(self) -> pathlib.Path:
 99        """
100        Return the target site-packages path for this virtual environment.
101        A `meerschaum.utils.venv.Venv` may have one virtual environment per minor Python version
102        (e.g. Python 3.10 and Python 3.7).
103        """
104        from meerschaum.utils.venv import venv_target_path
105        return venv_target_path(venv=self._venv, allow_nonexistent=True, debug=self._debug)

Return the target site-packages path for this virtual environment. A meerschaum.utils.venv.Venv may have one virtual environment per minor Python version (e.g. Python 3.10 and Python 3.7).

root_path: pathlib.Path
108    @property
109    def root_path(self) -> pathlib.Path:
110        """
111        Return the top-level path for this virtual environment.
112        """
113        import meerschaum.config.paths as paths
114        if self._venv is None:
115            return self.target_path.parent
116        return paths.VIRTENV_RESOURCES_PATH / self._venv

Return the top-level path for this virtual environment.

class Job:
  70class Job:
  71    """
  72    Manage a `meerschaum.utils.daemon.Daemon`, locally or remotely via the API.
  73    """
  74
  75    def __init__(
  76        self,
  77        name: str,
  78        sysargs: Union[List[str], str, None] = None,
  79        env: Optional[Dict[str, str]] = None,
  80        executor_keys: Optional[str] = None,
  81        delete_after_completion: bool = False,
  82        refresh_seconds: Union[int, float, None] = None,
  83        _properties: Optional[Dict[str, Any]] = None,
  84        _rotating_log=None,
  85        _stdin_file=None,
  86        _status_hook: Optional[Callable[[], str]] = None,
  87        _result_hook: Optional[Callable[[], SuccessTuple]] = None,
  88        _externally_managed: bool = False,
  89    ):
  90        """
  91        Create a new job to manage a `meerschaum.utils.daemon.Daemon`.
  92
  93        Parameters
  94        ----------
  95        name: str
  96            The name of the job to be created.
  97            This will also be used as the Daemon ID.
  98
  99        sysargs: Union[List[str], str, None], default None
 100            The sysargs of the command to be executed, e.g. 'start api'.
 101
 102        env: Optional[Dict[str, str]], default None
 103            If provided, set these environment variables in the job's process.
 104
 105        executor_keys: Optional[str], default None
 106            If provided, execute the job remotely on an API instance, e.g. 'api:main'.
 107
 108        delete_after_completion: bool, default False
 109            If `True`, delete this job when it has finished executing.
 110
 111        refresh_seconds: Union[int, float, None], default None
 112            The number of seconds to sleep between refreshes.
 113            Defaults to the configured value `system.cli.refresh_seconds`.
 114
 115        _properties: Optional[Dict[str, Any]], default None
 116            If provided, use this to patch the daemon's properties.
 117        """
 118        from meerschaum.utils.daemon import Daemon
 119        for char in BANNED_CHARS:
 120            if char in name:
 121                raise ValueError(f"Invalid name: ({char}) is not allowed.")
 122
 123        if isinstance(sysargs, str):
 124            sysargs = shlex.split(sysargs)
 125
 126        and_key = STATIC_CONFIG['system']['arguments']['and_key']
 127        escaped_and_key = STATIC_CONFIG['system']['arguments']['escaped_and_key']
 128        if sysargs:
 129            sysargs = [
 130                (arg if arg != escaped_and_key else and_key)
 131                for arg in sysargs
 132            ]
 133
 134        ### NOTE: 'local' and 'systemd' executors are being coalesced.
 135        if executor_keys is None:
 136            from meerschaum.jobs import get_executor_keys_from_context
 137            executor_keys = get_executor_keys_from_context()
 138
 139        self.executor_keys = executor_keys
 140        self.name = name
 141        self.refresh_seconds = (
 142            refresh_seconds
 143            if refresh_seconds is not None
 144            else mrsm.get_config('system', 'cli', 'refresh_seconds')
 145        )
 146        try:
 147            self._daemon = (
 148                Daemon(daemon_id=name)
 149                if executor_keys == 'local'
 150                else None
 151            )
 152        except Exception:
 153            self._daemon = None
 154
 155        ### Handle any injected dependencies.
 156        if _rotating_log is not None:
 157            self._rotating_log = _rotating_log
 158            if self._daemon is not None:
 159                self._daemon._rotating_log = _rotating_log
 160
 161        if _stdin_file is not None:
 162            self._stdin_file = _stdin_file
 163            if self._daemon is not None:
 164                self._daemon._stdin_file = _stdin_file
 165                self._daemon._blocking_stdin_file_path = _stdin_file.blocking_file_path
 166
 167        if _status_hook is not None:
 168            self._status_hook = _status_hook
 169
 170        if _result_hook is not None:
 171            self._result_hook = _result_hook
 172
 173        self._externally_managed = _externally_managed
 174        self._properties_patch = _properties or {}
 175        if _externally_managed:
 176            self._properties_patch.update({'externally_managed': _externally_managed})
 177
 178        if env:
 179            self._properties_patch.update({'env': env})
 180
 181        if delete_after_completion:
 182            self._properties_patch.update({'delete_after_completion': delete_after_completion})
 183
 184        daemon_sysargs = (
 185            self._daemon.properties.get('target', {}).get('args', [None])[0]
 186            if self._daemon is not None
 187            else None
 188        )
 189
 190        if daemon_sysargs and sysargs and daemon_sysargs != sysargs:
 191            warn("Given sysargs differ from existing sysargs.")
 192
 193        self._sysargs = [
 194            arg
 195            for arg in (daemon_sysargs or sysargs or [])
 196            if arg not in ('-d', '--daemon')
 197        ]
 198        for restart_flag in RESTART_FLAGS:
 199            if restart_flag in self._sysargs:
 200                self._properties_patch.update({'restart': True})
 201                break
 202
 203    @staticmethod
 204    def from_pid(pid: int, executor_keys: Optional[str] = None) -> Job:
 205        """
 206        Build a `Job` from the PID of a running Meerschaum process.
 207
 208        Parameters
 209        ----------
 210        pid: int
 211            The PID of the process.
 212
 213        executor_keys: Optional[str], default None
 214            The executor keys to assign to the job.
 215        """
 216        psutil = mrsm.attempt_import('psutil')
 217        try:
 218            process = psutil.Process(pid)
 219        except psutil.NoSuchProcess as e:
 220            warn(f"Process with PID {pid} does not exist.", stack=False)
 221            raise e
 222
 223        command_args = process.cmdline()
 224        is_daemon = command_args[1] == '-c'
 225
 226        if is_daemon:
 227            daemon_id = command_args[-1].split('daemon_id=')[-1].split(')')[0].replace("'", '')
 228            root_dir = process.environ().get(STATIC_CONFIG['environment']['root'], None)
 229            if root_dir is None:
 230                root_dir = paths.ROOT_DIR_PATH
 231            else:
 232                root_dir = pathlib.Path(root_dir)
 233            jobs_dir = root_dir / paths.DAEMON_RESOURCES_PATH.name
 234            daemon_dir = jobs_dir / daemon_id
 235            pid_file = daemon_dir / 'process.pid'
 236
 237            if pid_file.exists():
 238                with open(pid_file, 'r', encoding='utf-8') as f:
 239                    daemon_pid = int(f.read())
 240
 241                if pid != daemon_pid:
 242                    raise EnvironmentError(f"Differing PIDs: {pid=}, {daemon_pid=}")
 243            else:
 244                raise EnvironmentError(f"Is job '{daemon_id}' running?")
 245
 246            return Job(daemon_id, executor_keys=executor_keys)
 247
 248        from meerschaum._internal.arguments._parse_arguments import parse_arguments
 249        from meerschaum.utils.daemon import get_new_daemon_name
 250
 251        mrsm_ix = 0
 252        for i, arg in enumerate(command_args):
 253            if 'mrsm' in arg or 'meerschaum' in arg.lower():
 254                mrsm_ix = i
 255                break
 256
 257        sysargs = command_args[mrsm_ix+1:]
 258        kwargs = parse_arguments(sysargs)
 259        name = kwargs.get('name', get_new_daemon_name())
 260        return Job(name, sysargs, executor_keys=executor_keys)
 261
 262    def start(self, debug: bool = False) -> SuccessTuple:
 263        """
 264        Start the job's daemon.
 265        """
 266        if self.executor is not None:
 267            if not self.exists(debug=debug):
 268                return self.executor.create_job(
 269                    self.name,
 270                    self.sysargs,
 271                    properties=self.daemon.properties,
 272                    debug=debug,
 273                )
 274            return self.executor.start_job(self.name, debug=debug)
 275
 276        if self.is_running():
 277            return True, f"{self} is already running."
 278
 279        success, msg = self.daemon.run(
 280            keep_daemon_output=(not self.delete_after_completion),
 281            allow_dirty_run=True,
 282        )
 283        if not success:
 284            return success, msg
 285
 286        return success, f"Started {self}."
 287
 288    def stop(
 289        self,
 290        timeout_seconds: Union[int, float, None] = None,
 291        debug: bool = False,
 292    ) -> SuccessTuple:
 293        """
 294        Stop the job's daemon.
 295        """
 296        if self.executor is not None:
 297            return self.executor.stop_job(self.name, debug=debug)
 298
 299        if self.daemon.status == 'stopped':
 300            if not self.restart:
 301                return True, f"{self} is not running."
 302            elif self.stop_time is not None:
 303                return True, f"{self} will not restart until manually started."
 304
 305        quit_success, quit_msg = self.daemon.quit(timeout=timeout_seconds)
 306        if quit_success:
 307            return quit_success, f"Stopped {self}."
 308
 309        warn(
 310            f"Failed to gracefully quit {self}: {quit_msg}",
 311            stack=False,
 312        )
 313        kill_success, kill_msg = self.daemon.kill(timeout=timeout_seconds)
 314        if not kill_success:
 315            return kill_success, kill_msg
 316
 317        return kill_success, f"Killed {self}."
 318
 319    def pause(
 320        self,
 321        timeout_seconds: Union[int, float, None] = None,
 322        debug: bool = False,
 323    ) -> SuccessTuple:
 324        """
 325        Pause the job's daemon.
 326        """
 327        if self.executor is not None:
 328            return self.executor.pause_job(self.name, debug=debug)
 329
 330        pause_success, pause_msg = self.daemon.pause(timeout=timeout_seconds)
 331        if not pause_success:
 332            return pause_success, pause_msg
 333
 334        return pause_success, f"Paused {self}."
 335
 336    def delete(self, debug: bool = False) -> SuccessTuple:
 337        """
 338        Delete the job and its daemon.
 339        """
 340        if self.executor is not None:
 341            return self.executor.delete_job(self.name, debug=debug)
 342
 343        if self.is_running():
 344            stop_success, stop_msg = self.stop()
 345            if not stop_success:
 346                return stop_success, stop_msg
 347
 348        cleanup_success, cleanup_msg = self.daemon.cleanup()
 349        if not cleanup_success:
 350            return cleanup_success, cleanup_msg
 351
 352        _ = self.daemon._properties.pop('result', None)
 353        return cleanup_success, f"Deleted {self}."
 354
 355    def is_running(self) -> bool:
 356        """
 357        Determine whether the job's daemon is running.
 358        """
 359        return self.status == 'running'
 360
 361    def exists(self, debug: bool = False) -> bool:
 362        """
 363        Determine whether the job exists.
 364        """
 365        if self.executor is not None:
 366            return self.executor.get_job_exists(self.name, debug=debug)
 367
 368        return self.daemon.path.exists()
 369
 370    def get_logs(self) -> Union[str, None]:
 371        """
 372        Return the output text of the job's daemon.
 373        """
 374        if self.executor is not None:
 375            return self.executor.get_logs(self.name)
 376
 377        return self.daemon.log_text
 378
 379    def monitor_logs(
 380        self,
 381        callback_function: Callable[[str], None] = _default_stdout_callback,
 382        input_callback_function: Optional[Callable[[], str]] = None,
 383        stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None,
 384        stop_event: Optional[asyncio.Event] = None,
 385        stop_on_exit: bool = False,
 386        strip_timestamps: bool = False,
 387        accept_input: bool = True,
 388        debug: bool = False,
 389        _logs_path: Optional[pathlib.Path] = None,
 390        _log=None,
 391        _stdin_file=None,
 392        _wait_if_stopped: bool = True,
 393    ):
 394        """
 395        Monitor the job's log files and execute a callback on new lines.
 396
 397        Parameters
 398        ----------
 399        callback_function: Callable[[str], None], default partial(print, end='')
 400            The callback to execute as new data comes in.
 401            Defaults to printing the output directly to `stdout`.
 402
 403        input_callback_function: Optional[Callable[[], str]], default None
 404            If provided, execute this callback when the daemon is blocking on stdin.
 405            Defaults to `sys.stdin.readline()`.
 406
 407        stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None
 408            If provided, execute this callback when the daemon stops.
 409            The job's SuccessTuple will be passed to the callback.
 410
 411        stop_event: Optional[asyncio.Event], default None
 412            If provided, stop monitoring when this event is set.
 413            You may instead raise `meerschaum.jobs.StopMonitoringLogs`
 414            from within `callback_function` to stop monitoring.
 415
 416        stop_on_exit: bool, default False
 417            If `True`, stop monitoring when the job stops.
 418
 419        strip_timestamps: bool, default False
 420            If `True`, remove leading timestamps from lines.
 421
 422        accept_input: bool, default True
 423            If `True`, accept input when the daemon blocks on stdin.
 424        """
 425        if self.executor is not None:
 426            self.executor.monitor_logs(
 427                self.name,
 428                callback_function,
 429                input_callback_function=input_callback_function,
 430                stop_callback_function=stop_callback_function,
 431                stop_on_exit=stop_on_exit,
 432                accept_input=accept_input,
 433                strip_timestamps=strip_timestamps,
 434                debug=debug,
 435            )
 436            return
 437
 438        monitor_logs_coroutine = self.monitor_logs_async(
 439            callback_function=callback_function,
 440            input_callback_function=input_callback_function,
 441            stop_callback_function=stop_callback_function,
 442            stop_event=stop_event,
 443            stop_on_exit=stop_on_exit,
 444            strip_timestamps=strip_timestamps,
 445            accept_input=accept_input,
 446            debug=debug,
 447            _logs_path=_logs_path,
 448            _log=_log,
 449            _stdin_file=_stdin_file,
 450            _wait_if_stopped=_wait_if_stopped,
 451        )
 452        return asyncio.run(monitor_logs_coroutine)
 453
 454    async def monitor_logs_async(
 455        self,
 456        callback_function: Callable[[str], None] = _default_stdout_callback,
 457        input_callback_function: Optional[Callable[[], str]] = None,
 458        stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None,
 459        stop_event: Optional[asyncio.Event] = None,
 460        stop_on_exit: bool = False,
 461        strip_timestamps: bool = False,
 462        accept_input: bool = True,
 463        debug: bool = False,
 464        _logs_path: Optional[pathlib.Path] = None,
 465        _log=None,
 466        _stdin_file=None,
 467        _wait_if_stopped: bool = True,
 468    ):
 469        """
 470        Monitor the job's log files and await a callback on new lines.
 471
 472        Parameters
 473        ----------
 474        callback_function: Callable[[str], None], default _default_stdout_callback
 475            The callback to execute as new data comes in.
 476            Defaults to printing the output directly to `stdout`.
 477
 478        input_callback_function: Optional[Callable[[], str]], default None
 479            If provided, execute this callback when the daemon is blocking on stdin.
 480            Defaults to `sys.stdin.readline()`.
 481
 482        stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None
 483            If provided, execute this callback when the daemon stops.
 484            The job's SuccessTuple will be passed to the callback.
 485
 486        stop_event: Optional[asyncio.Event], default None
 487            If provided, stop monitoring when this event is set.
 488            You may instead raise `meerschaum.jobs.StopMonitoringLogs`
 489            from within `callback_function` to stop monitoring.
 490
 491        stop_on_exit: bool, default False
 492            If `True`, stop monitoring when the job stops.
 493
 494        strip_timestamps: bool, default False
 495            If `True`, remove leading timestamps from lines.
 496
 497        accept_input: bool, default True
 498            If `True`, accept input when the daemon blocks on stdin.
 499        """
 500        from meerschaum.utils.prompt import prompt
 501
 502        def default_input_callback_function():
 503            prompt_kwargs = self.get_prompt_kwargs(debug=debug)
 504            if prompt_kwargs:
 505                answer = prompt(**prompt_kwargs)
 506                return answer + '\n'
 507            return sys.stdin.readline()
 508
 509        if input_callback_function is None:
 510            input_callback_function = default_input_callback_function
 511
 512        if self.executor is not None:
 513            await self.executor.monitor_logs_async(
 514                self.name,
 515                callback_function,
 516                input_callback_function=input_callback_function,
 517                stop_callback_function=stop_callback_function,
 518                stop_on_exit=stop_on_exit,
 519                strip_timestamps=strip_timestamps,
 520                accept_input=accept_input,
 521                debug=debug,
 522            )
 523            return
 524
 525        from meerschaum.utils.formatting._jobs import strip_timestamp_from_line
 526
 527        events = {
 528            'user': stop_event,
 529            'stopped': asyncio.Event(),
 530            'stop_token': asyncio.Event(),
 531            'stop_exception': asyncio.Event(),
 532            'stopped_timeout': asyncio.Event(),
 533        }
 534        combined_event = asyncio.Event()
 535        emitted_text = False
 536        stdin_file = _stdin_file if _stdin_file is not None else self.daemon.stdin_file
 537
 538        async def check_job_status():
 539            if not stop_on_exit:
 540                return
 541
 542            nonlocal emitted_text
 543
 544            sleep_time = 0.1
 545            while sleep_time < 0.2:
 546                if self.status == 'stopped':
 547                    if not emitted_text and _wait_if_stopped:
 548                        await asyncio.sleep(sleep_time)
 549                        sleep_time = round(sleep_time * 1.1, 3)
 550                        continue
 551
 552                    if stop_callback_function is not None:
 553                        try:
 554                            if asyncio.iscoroutinefunction(stop_callback_function):
 555                                await stop_callback_function(self.result)
 556                            else:
 557                                stop_callback_function(self.result)
 558                        except asyncio.exceptions.CancelledError:
 559                            break
 560                        except Exception:
 561                            warn(traceback.format_exc())
 562
 563                    if stop_on_exit:
 564                        events['stopped'].set()
 565
 566                    break
 567                await asyncio.sleep(0.1)
 568
 569            events['stopped_timeout'].set()
 570
 571        async def check_blocking_on_input():
 572            while True:
 573                if not emitted_text or not self.is_blocking_on_stdin():
 574                    try:
 575                        await asyncio.sleep(self.refresh_seconds)
 576                    except asyncio.exceptions.CancelledError:
 577                        break
 578                    continue
 579
 580                if not self.is_running():
 581                    break
 582
 583                await emit_latest_lines()
 584
 585                try:
 586                    print('', end='', flush=True)
 587                    if asyncio.iscoroutinefunction(input_callback_function):
 588                        data = await input_callback_function()
 589                    else:
 590                        loop = asyncio.get_running_loop()
 591                        data = await loop.run_in_executor(None, input_callback_function)
 592                except KeyboardInterrupt:
 593                    break
 594                #  if not data.endswith('\n'):
 595                    #  data += '\n'
 596
 597                stdin_file.write(data)
 598                await asyncio.sleep(self.refresh_seconds)
 599
 600        async def combine_events():
 601            event_tasks = [
 602                asyncio.create_task(event.wait())
 603                for event in events.values()
 604                if event is not None
 605            ]
 606            if not event_tasks:
 607                return
 608
 609            try:
 610                done, pending = await asyncio.wait(
 611                    event_tasks,
 612                    return_when=asyncio.FIRST_COMPLETED,
 613                )
 614                for task in pending:
 615                    task.cancel()
 616            except asyncio.exceptions.CancelledError:
 617                pass
 618            finally:
 619                combined_event.set()
 620
 621        check_job_status_task = asyncio.create_task(check_job_status())
 622        check_blocking_on_input_task = asyncio.create_task(check_blocking_on_input())
 623        combine_events_task = asyncio.create_task(combine_events())
 624
 625        log = _log if _log is not None else self.daemon.rotating_log
 626        lines_to_show = (
 627            self.daemon.properties.get(
 628                'logs', {}
 629            ).get(
 630                'lines_to_show', get_config('jobs', 'logs', 'lines_to_show')
 631            )
 632        )
 633
 634        async def emit_latest_lines():
 635            nonlocal emitted_text
 636            nonlocal stop_event
 637            lines = log.readlines()
 638            for line in lines[(-1 * lines_to_show):]:
 639                if stop_event is not None and stop_event.is_set():
 640                    return
 641
 642                line_stripped_extra = strip_timestamp_from_line(line.strip())
 643                line_stripped = strip_timestamp_from_line(line)
 644
 645                if line_stripped_extra == STOP_TOKEN:
 646                    events['stop_token'].set()
 647                    return
 648
 649                if line_stripped_extra == CLEAR_TOKEN:
 650                    clear_screen(debug=debug)
 651                    continue
 652
 653                if line_stripped_extra == FLUSH_TOKEN.strip():
 654                    line_stripped = ''
 655                    line = ''
 656
 657                if strip_timestamps:
 658                    line = line_stripped
 659
 660                try:
 661                    if asyncio.iscoroutinefunction(callback_function):
 662                        await callback_function(line)
 663                    else:
 664                        callback_function(line)
 665                    emitted_text = True
 666                except StopMonitoringLogs:
 667                    events['stop_exception'].set()
 668                    return
 669                except Exception:
 670                    warn(f"Error in logs callback:\n{traceback.format_exc()}")
 671
 672        await emit_latest_lines()
 673
 674        tasks = (
 675            [check_job_status_task]
 676            + ([check_blocking_on_input_task] if accept_input else [])
 677            + [combine_events_task]
 678        )
 679        try:
 680            _ = asyncio.gather(*tasks, return_exceptions=True)
 681        except asyncio.exceptions.CancelledError:
 682            raise
 683        except Exception:
 684            warn(f"Failed to run async checks:\n{traceback.format_exc()}")
 685
 686        watchfiles = mrsm.attempt_import('watchfiles', lazy=False)
 687        dir_path_to_monitor = (
 688            _logs_path
 689            or (log.file_path.parent if log else None)
 690            or paths.LOGS_RESOURCES_PATH
 691        )
 692        async for changes in watchfiles.awatch(
 693            dir_path_to_monitor,
 694            stop_event=combined_event,
 695        ):
 696            for change in changes:
 697                file_path_str = change[1]
 698                file_path = pathlib.Path(file_path_str)
 699                latest_subfile_path = log.get_latest_subfile_path()
 700                if latest_subfile_path != file_path:
 701                    continue
 702
 703                await emit_latest_lines()
 704
 705        await emit_latest_lines()
 706
 707    def is_blocking_on_stdin(self, debug: bool = False) -> bool:
 708        """
 709        Return whether a job's daemon is blocking on stdin.
 710        """
 711        if self.executor is not None:
 712            return self.executor.get_job_is_blocking_on_stdin(self.name, debug=debug)
 713
 714        return self.is_running() and self.daemon.blocking_stdin_file_path.exists()
 715
 716    def get_prompt_kwargs(self, debug: bool = False) -> Dict[str, Any]:
 717        """
 718        Return the kwargs to the blocking `prompt()`, if available.
 719        """
 720        if self.executor is not None:
 721            return self.executor.get_job_prompt_kwargs(self.name, debug=debug)
 722
 723        if not self.daemon.prompt_kwargs_file_path.exists():
 724            return {}
 725
 726        try:
 727            with open(self.daemon.prompt_kwargs_file_path, 'r', encoding='utf-8') as f:
 728                prompt_kwargs = json.load(f)
 729
 730            return prompt_kwargs
 731        
 732        except Exception:
 733            import traceback
 734            traceback.print_exc()
 735            return {}
 736
 737    def write_stdin(self, data):
 738        """
 739        Write to a job's daemon's `stdin`.
 740        """
 741        self.daemon.stdin_file.write(data)
 742
 743    @property
 744    def executor(self) -> Union[Executor, None]:
 745        """
 746        If the job is remote, return the connector to the remote API instance.
 747        """
 748        return (
 749            mrsm.get_connector(self.executor_keys)
 750            if self.executor_keys != 'local'
 751            else None
 752        )
 753
 754    @property
 755    def status(self) -> str:
 756        """
 757        Return the running status of the job's daemon.
 758        """
 759        if '_status_hook' in self.__dict__:
 760            return self._status_hook()
 761
 762        if self.executor is not None:
 763            return self.executor.get_job_status(self.name)
 764
 765        return self.daemon.status
 766
 767    @property
 768    def pid(self) -> Union[int, None]:
 769        """
 770        Return the PID of the job's dameon.
 771        """
 772        if self.executor is not None:
 773            return self.executor.get_job_metadata(self.name).get('daemon', {}).get('pid', None)
 774
 775        return self.daemon.pid
 776
 777    @property
 778    def restart(self) -> bool:
 779        """
 780        Return whether to restart a stopped job.
 781        """
 782        if self.executor is not None:
 783            return self.executor.get_job_metadata(self.name).get('restart', False)
 784
 785        return self.daemon.properties.get('restart', False)
 786
 787    @property
 788    def result(self) -> SuccessTuple:
 789        """
 790        Return the `SuccessTuple` when the job has terminated.
 791        """
 792        if self.is_running():
 793            return True, f"{self} is running."
 794
 795        if '_result_hook' in self.__dict__:
 796            return self._result_hook()
 797
 798        if self.executor is not None:
 799            return (
 800                self.executor.get_job_metadata(self.name)
 801                .get('result', (False, "No result available."))
 802            )
 803
 804        _result = self.daemon.properties.get('result', None)
 805        if _result is None:
 806            from meerschaum.utils.daemon.Daemon import _results
 807            return _results.get(self.daemon.daemon_id, (False, "No result available."))
 808
 809        return tuple(_result)
 810
 811    @property
 812    def sysargs(self) -> List[str]:
 813        """
 814        Return the sysargs to use for the Daemon.
 815        """
 816        if self._sysargs:
 817            return self._sysargs
 818
 819        if self.executor is not None:
 820            return self.executor.get_job_metadata(self.name).get('sysargs', [])
 821
 822        target_args = self.daemon.target_args
 823        if target_args is None:
 824            return []
 825        self._sysargs = target_args[0] if len(target_args) > 0 else []
 826        return self._sysargs
 827
 828    def get_daemon_properties(self) -> Dict[str, Any]:
 829        """
 830        Return the `properties` dictionary for the job's daemon.
 831        """
 832        remote_properties = (
 833            {}
 834            if self.executor is None
 835            else self.executor.get_job_properties(self.name)
 836        )
 837        return {
 838            **remote_properties,
 839            **self._properties_patch
 840        }
 841
 842    @property
 843    def daemon(self) -> 'Daemon':
 844        """
 845        Return the daemon which this job manages.
 846        """
 847        from meerschaum.utils.daemon import Daemon
 848        if self._daemon is not None and self.executor is None and self._sysargs:
 849            return self._daemon
 850
 851        self._daemon = Daemon(
 852            target=entry,
 853            target_args=[self._sysargs],
 854            target_kw={},
 855            daemon_id=self.name,
 856            label=shlex.join(self._sysargs),
 857            properties=self.get_daemon_properties(),
 858        )
 859        if '_rotating_log' in self.__dict__:
 860            self._daemon._rotating_log = self._rotating_log
 861
 862        if '_stdin_file' in self.__dict__:
 863            self._daemon._stdin_file = self._stdin_file
 864            self._daemon._blocking_stdin_file_path = self._stdin_file.blocking_file_path
 865
 866        return self._daemon
 867
 868    @property
 869    def began(self) -> Union[datetime, None]:
 870        """
 871        The datetime when the job began running.
 872        """
 873        if self.executor is not None:
 874            began_str = self.executor.get_job_began(self.name)
 875            if began_str is None:
 876                return None
 877            return (
 878                datetime.fromisoformat(began_str)
 879                .astimezone(timezone.utc)
 880                .replace(tzinfo=None)
 881            )
 882
 883        began_str = self.daemon.properties.get('process', {}).get('began', None)
 884        if began_str is None:
 885            return None
 886
 887        return datetime.fromisoformat(began_str)
 888
 889    @property
 890    def ended(self) -> Union[datetime, None]:
 891        """
 892        The datetime when the job stopped running.
 893        """
 894        if self.executor is not None:
 895            ended_str = self.executor.get_job_ended(self.name)
 896            if ended_str is None:
 897                return None
 898            return (
 899                datetime.fromisoformat(ended_str)
 900                .astimezone(timezone.utc)
 901                .replace(tzinfo=None)
 902            )
 903
 904        ended_str = self.daemon.properties.get('process', {}).get('ended', None)
 905        if ended_str is None:
 906            return None
 907
 908        return datetime.fromisoformat(ended_str)
 909
 910    @property
 911    def paused(self) -> Union[datetime, None]:
 912        """
 913        The datetime when the job was suspended while running.
 914        """
 915        if self.executor is not None:
 916            paused_str = self.executor.get_job_paused(self.name)
 917            if paused_str is None:
 918                return None
 919            return (
 920                datetime.fromisoformat(paused_str)
 921                .astimezone(timezone.utc)
 922                .replace(tzinfo=None)
 923            )
 924
 925        paused_str = self.daemon.properties.get('process', {}).get('paused', None)
 926        if paused_str is None:
 927            return None
 928
 929        return datetime.fromisoformat(paused_str)
 930
 931    @property
 932    def stop_time(self) -> Union[datetime, None]:
 933        """
 934        Return the timestamp when the job was manually stopped.
 935        """
 936        if self.executor is not None:
 937            return self.executor.get_job_stop_time(self.name)
 938
 939        if not self.daemon.stop_path.exists():
 940            return None
 941
 942        stop_data = self.daemon._read_stop_file()
 943        if not stop_data:
 944            return None
 945
 946        stop_time_str = stop_data.get('stop_time', None)
 947        if not stop_time_str:
 948            warn(f"Could not read stop time for {self}.")
 949            return None
 950
 951        return datetime.fromisoformat(stop_time_str)
 952
 953    @property
 954    def hidden(self) -> bool:
 955        """
 956        Return a bool indicating whether this job should be displayed.
 957        """
 958        return (
 959            self.name.startswith('_')
 960            or self.name.startswith('.')
 961            or self._is_externally_managed
 962        )
 963
 964    def check_restart(self) -> SuccessTuple:
 965        """
 966        If `restart` is `True` and the daemon is not running,
 967        restart the job.
 968        Do not restart if the job was manually stopped.
 969        """
 970        if self.is_running():
 971            return True, f"{self} is running."
 972
 973        if not self.restart:
 974            return True, f"{self} does not need to be restarted."
 975
 976        if self.stop_time is not None:
 977            return True, f"{self} was manually stopped."
 978
 979        return self.start()
 980
 981    @property
 982    def label(self) -> str:
 983        """
 984        Return the job's Daemon label (joined sysargs).
 985        """
 986        from meerschaum._internal.arguments import compress_pipeline_sysargs
 987        sysargs = compress_pipeline_sysargs(self.sysargs)
 988        return shlex.join(sysargs).replace(' + ', '\n+ ').replace(' : ', '\n: ').lstrip().rstrip()
 989
 990    @property
 991    def _externally_managed_file(self) -> pathlib.Path:
 992        """
 993        Return the path to the externally managed file.
 994        """
 995        return self.daemon.path / '.externally-managed'
 996
 997    def _set_externally_managed(self):
 998        """
 999        Set this job as externally managed.
1000        """
1001        self._externally_managed = True
1002        try:
1003            self._externally_managed_file.parent.mkdir(exist_ok=True, parents=True)
1004            self._externally_managed_file.touch()
1005        except Exception as e:
1006            warn(e)
1007
1008    @property
1009    def _is_externally_managed(self) -> bool:
1010        """
1011        Return whether this job is externally managed.
1012        """
1013        return self.executor_keys in (None, 'local') and (
1014            self._externally_managed or self._externally_managed_file.exists()
1015        )
1016
1017    @property
1018    def env(self) -> Dict[str, str]:
1019        """
1020        Return the environment variables to set for the job's process.
1021        """
1022        if '_env' in self.__dict__:
1023            return self.__dict__['_env']
1024
1025        _env = self.daemon.properties.get('env', {})
1026        default_env = {
1027            'PYTHONUNBUFFERED': '1',
1028            'LINES': str(get_config('jobs', 'terminal', 'lines')),
1029            'COLUMNS': str(get_config('jobs', 'terminal', 'columns')),
1030            STATIC_CONFIG['environment']['noninteractive']: 'true',
1031        }
1032        self._env = {**default_env, **_env}
1033        return self._env
1034
1035    @property
1036    def delete_after_completion(self) -> bool:
1037        """
1038        Return whether this job is configured to delete itself after completion.
1039        """
1040        if '_delete_after_completion' in self.__dict__:
1041            return self.__dict__.get('_delete_after_completion', False)
1042
1043        self._delete_after_completion = self.daemon.properties.get('delete_after_completion', False)
1044        return self._delete_after_completion
1045
1046    def __str__(self) -> str:
1047        sysargs = self.sysargs
1048        sysargs_str = shlex.join(sysargs) if sysargs else ''
1049        job_str = f'Job("{self.name}"'
1050        if sysargs_str:
1051            job_str += f', "{sysargs_str}"'
1052
1053        job_str += ')'
1054        return job_str
1055
1056    def __repr__(self) -> str:
1057        return str(self)
1058
1059    def __hash__(self) -> int:
1060        return hash(self.name)

Manage a meerschaum.utils.daemon.Daemon, locally or remotely via the API.

Job( name: str, sysargs: Union[List[str], str, NoneType] = None, env: Optional[Dict[str, str]] = None, executor_keys: Optional[str] = None, delete_after_completion: bool = False, refresh_seconds: Union[int, float, NoneType] = None, _properties: Optional[Dict[str, Any]] = None, _rotating_log=None, _stdin_file=None, _status_hook: Optional[Callable[[], str]] = None, _result_hook: Optional[Callable[[], Tuple[bool, str]]] = None, _externally_managed: bool = False)
 75    def __init__(
 76        self,
 77        name: str,
 78        sysargs: Union[List[str], str, None] = None,
 79        env: Optional[Dict[str, str]] = None,
 80        executor_keys: Optional[str] = None,
 81        delete_after_completion: bool = False,
 82        refresh_seconds: Union[int, float, None] = None,
 83        _properties: Optional[Dict[str, Any]] = None,
 84        _rotating_log=None,
 85        _stdin_file=None,
 86        _status_hook: Optional[Callable[[], str]] = None,
 87        _result_hook: Optional[Callable[[], SuccessTuple]] = None,
 88        _externally_managed: bool = False,
 89    ):
 90        """
 91        Create a new job to manage a `meerschaum.utils.daemon.Daemon`.
 92
 93        Parameters
 94        ----------
 95        name: str
 96            The name of the job to be created.
 97            This will also be used as the Daemon ID.
 98
 99        sysargs: Union[List[str], str, None], default None
100            The sysargs of the command to be executed, e.g. 'start api'.
101
102        env: Optional[Dict[str, str]], default None
103            If provided, set these environment variables in the job's process.
104
105        executor_keys: Optional[str], default None
106            If provided, execute the job remotely on an API instance, e.g. 'api:main'.
107
108        delete_after_completion: bool, default False
109            If `True`, delete this job when it has finished executing.
110
111        refresh_seconds: Union[int, float, None], default None
112            The number of seconds to sleep between refreshes.
113            Defaults to the configured value `system.cli.refresh_seconds`.
114
115        _properties: Optional[Dict[str, Any]], default None
116            If provided, use this to patch the daemon's properties.
117        """
118        from meerschaum.utils.daemon import Daemon
119        for char in BANNED_CHARS:
120            if char in name:
121                raise ValueError(f"Invalid name: ({char}) is not allowed.")
122
123        if isinstance(sysargs, str):
124            sysargs = shlex.split(sysargs)
125
126        and_key = STATIC_CONFIG['system']['arguments']['and_key']
127        escaped_and_key = STATIC_CONFIG['system']['arguments']['escaped_and_key']
128        if sysargs:
129            sysargs = [
130                (arg if arg != escaped_and_key else and_key)
131                for arg in sysargs
132            ]
133
134        ### NOTE: 'local' and 'systemd' executors are being coalesced.
135        if executor_keys is None:
136            from meerschaum.jobs import get_executor_keys_from_context
137            executor_keys = get_executor_keys_from_context()
138
139        self.executor_keys = executor_keys
140        self.name = name
141        self.refresh_seconds = (
142            refresh_seconds
143            if refresh_seconds is not None
144            else mrsm.get_config('system', 'cli', 'refresh_seconds')
145        )
146        try:
147            self._daemon = (
148                Daemon(daemon_id=name)
149                if executor_keys == 'local'
150                else None
151            )
152        except Exception:
153            self._daemon = None
154
155        ### Handle any injected dependencies.
156        if _rotating_log is not None:
157            self._rotating_log = _rotating_log
158            if self._daemon is not None:
159                self._daemon._rotating_log = _rotating_log
160
161        if _stdin_file is not None:
162            self._stdin_file = _stdin_file
163            if self._daemon is not None:
164                self._daemon._stdin_file = _stdin_file
165                self._daemon._blocking_stdin_file_path = _stdin_file.blocking_file_path
166
167        if _status_hook is not None:
168            self._status_hook = _status_hook
169
170        if _result_hook is not None:
171            self._result_hook = _result_hook
172
173        self._externally_managed = _externally_managed
174        self._properties_patch = _properties or {}
175        if _externally_managed:
176            self._properties_patch.update({'externally_managed': _externally_managed})
177
178        if env:
179            self._properties_patch.update({'env': env})
180
181        if delete_after_completion:
182            self._properties_patch.update({'delete_after_completion': delete_after_completion})
183
184        daemon_sysargs = (
185            self._daemon.properties.get('target', {}).get('args', [None])[0]
186            if self._daemon is not None
187            else None
188        )
189
190        if daemon_sysargs and sysargs and daemon_sysargs != sysargs:
191            warn("Given sysargs differ from existing sysargs.")
192
193        self._sysargs = [
194            arg
195            for arg in (daemon_sysargs or sysargs or [])
196            if arg not in ('-d', '--daemon')
197        ]
198        for restart_flag in RESTART_FLAGS:
199            if restart_flag in self._sysargs:
200                self._properties_patch.update({'restart': True})
201                break

Create a new job to manage a meerschaum.utils.daemon.Daemon.

Parameters
  • name (str): The name of the job to be created. This will also be used as the Daemon ID.
  • sysargs (Union[List[str], str, None], default None): The sysargs of the command to be executed, e.g. 'start api'.
  • env (Optional[Dict[str, str]], default None): If provided, set these environment variables in the job's process.
  • executor_keys (Optional[str], default None): If provided, execute the job remotely on an API instance, e.g. 'api:main'.
  • delete_after_completion (bool, default False): If True, delete this job when it has finished executing.
  • refresh_seconds (Union[int, float, None], default None): The number of seconds to sleep between refreshes. Defaults to the configured value system.cli.refresh_seconds.
  • _properties (Optional[Dict[str, Any]], default None): If provided, use this to patch the daemon's properties.
executor_keys
name
refresh_seconds
@staticmethod
def from_pid( pid: int, executor_keys: Optional[str] = None) -> Job:
203    @staticmethod
204    def from_pid(pid: int, executor_keys: Optional[str] = None) -> Job:
205        """
206        Build a `Job` from the PID of a running Meerschaum process.
207
208        Parameters
209        ----------
210        pid: int
211            The PID of the process.
212
213        executor_keys: Optional[str], default None
214            The executor keys to assign to the job.
215        """
216        psutil = mrsm.attempt_import('psutil')
217        try:
218            process = psutil.Process(pid)
219        except psutil.NoSuchProcess as e:
220            warn(f"Process with PID {pid} does not exist.", stack=False)
221            raise e
222
223        command_args = process.cmdline()
224        is_daemon = command_args[1] == '-c'
225
226        if is_daemon:
227            daemon_id = command_args[-1].split('daemon_id=')[-1].split(')')[0].replace("'", '')
228            root_dir = process.environ().get(STATIC_CONFIG['environment']['root'], None)
229            if root_dir is None:
230                root_dir = paths.ROOT_DIR_PATH
231            else:
232                root_dir = pathlib.Path(root_dir)
233            jobs_dir = root_dir / paths.DAEMON_RESOURCES_PATH.name
234            daemon_dir = jobs_dir / daemon_id
235            pid_file = daemon_dir / 'process.pid'
236
237            if pid_file.exists():
238                with open(pid_file, 'r', encoding='utf-8') as f:
239                    daemon_pid = int(f.read())
240
241                if pid != daemon_pid:
242                    raise EnvironmentError(f"Differing PIDs: {pid=}, {daemon_pid=}")
243            else:
244                raise EnvironmentError(f"Is job '{daemon_id}' running?")
245
246            return Job(daemon_id, executor_keys=executor_keys)
247
248        from meerschaum._internal.arguments._parse_arguments import parse_arguments
249        from meerschaum.utils.daemon import get_new_daemon_name
250
251        mrsm_ix = 0
252        for i, arg in enumerate(command_args):
253            if 'mrsm' in arg or 'meerschaum' in arg.lower():
254                mrsm_ix = i
255                break
256
257        sysargs = command_args[mrsm_ix+1:]
258        kwargs = parse_arguments(sysargs)
259        name = kwargs.get('name', get_new_daemon_name())
260        return Job(name, sysargs, executor_keys=executor_keys)

Build a Job from the PID of a running Meerschaum process.

Parameters
  • pid (int): The PID of the process.
  • executor_keys (Optional[str], default None): The executor keys to assign to the job.
def start(self, debug: bool = False) -> Tuple[bool, str]:
262    def start(self, debug: bool = False) -> SuccessTuple:
263        """
264        Start the job's daemon.
265        """
266        if self.executor is not None:
267            if not self.exists(debug=debug):
268                return self.executor.create_job(
269                    self.name,
270                    self.sysargs,
271                    properties=self.daemon.properties,
272                    debug=debug,
273                )
274            return self.executor.start_job(self.name, debug=debug)
275
276        if self.is_running():
277            return True, f"{self} is already running."
278
279        success, msg = self.daemon.run(
280            keep_daemon_output=(not self.delete_after_completion),
281            allow_dirty_run=True,
282        )
283        if not success:
284            return success, msg
285
286        return success, f"Started {self}."

Start the job's daemon.

def stop( self, timeout_seconds: Union[int, float, NoneType] = None, debug: bool = False) -> Tuple[bool, str]:
288    def stop(
289        self,
290        timeout_seconds: Union[int, float, None] = None,
291        debug: bool = False,
292    ) -> SuccessTuple:
293        """
294        Stop the job's daemon.
295        """
296        if self.executor is not None:
297            return self.executor.stop_job(self.name, debug=debug)
298
299        if self.daemon.status == 'stopped':
300            if not self.restart:
301                return True, f"{self} is not running."
302            elif self.stop_time is not None:
303                return True, f"{self} will not restart until manually started."
304
305        quit_success, quit_msg = self.daemon.quit(timeout=timeout_seconds)
306        if quit_success:
307            return quit_success, f"Stopped {self}."
308
309        warn(
310            f"Failed to gracefully quit {self}: {quit_msg}",
311            stack=False,
312        )
313        kill_success, kill_msg = self.daemon.kill(timeout=timeout_seconds)
314        if not kill_success:
315            return kill_success, kill_msg
316
317        return kill_success, f"Killed {self}."

Stop the job's daemon.

def pause( self, timeout_seconds: Union[int, float, NoneType] = None, debug: bool = False) -> Tuple[bool, str]:
319    def pause(
320        self,
321        timeout_seconds: Union[int, float, None] = None,
322        debug: bool = False,
323    ) -> SuccessTuple:
324        """
325        Pause the job's daemon.
326        """
327        if self.executor is not None:
328            return self.executor.pause_job(self.name, debug=debug)
329
330        pause_success, pause_msg = self.daemon.pause(timeout=timeout_seconds)
331        if not pause_success:
332            return pause_success, pause_msg
333
334        return pause_success, f"Paused {self}."

Pause the job's daemon.

def delete(self, debug: bool = False) -> Tuple[bool, str]:
336    def delete(self, debug: bool = False) -> SuccessTuple:
337        """
338        Delete the job and its daemon.
339        """
340        if self.executor is not None:
341            return self.executor.delete_job(self.name, debug=debug)
342
343        if self.is_running():
344            stop_success, stop_msg = self.stop()
345            if not stop_success:
346                return stop_success, stop_msg
347
348        cleanup_success, cleanup_msg = self.daemon.cleanup()
349        if not cleanup_success:
350            return cleanup_success, cleanup_msg
351
352        _ = self.daemon._properties.pop('result', None)
353        return cleanup_success, f"Deleted {self}."

Delete the job and its daemon.

def is_running(self) -> bool:
355    def is_running(self) -> bool:
356        """
357        Determine whether the job's daemon is running.
358        """
359        return self.status == 'running'

Determine whether the job's daemon is running.

def exists(self, debug: bool = False) -> bool:
361    def exists(self, debug: bool = False) -> bool:
362        """
363        Determine whether the job exists.
364        """
365        if self.executor is not None:
366            return self.executor.get_job_exists(self.name, debug=debug)
367
368        return self.daemon.path.exists()

Determine whether the job exists.

def get_logs(self) -> Optional[str]:
370    def get_logs(self) -> Union[str, None]:
371        """
372        Return the output text of the job's daemon.
373        """
374        if self.executor is not None:
375            return self.executor.get_logs(self.name)
376
377        return self.daemon.log_text

Return the output text of the job's daemon.

def monitor_logs( self, callback_function: Callable[[str], NoneType] = <function _default_stdout_callback>, input_callback_function: Optional[Callable[[], str]] = None, stop_callback_function: Optional[Callable[[Tuple[bool, str]], NoneType]] = None, stop_event: Optional[asyncio.locks.Event] = None, stop_on_exit: bool = False, strip_timestamps: bool = False, accept_input: bool = True, debug: bool = False, _logs_path: Optional[pathlib.Path] = None, _log=None, _stdin_file=None, _wait_if_stopped: bool = True):
379    def monitor_logs(
380        self,
381        callback_function: Callable[[str], None] = _default_stdout_callback,
382        input_callback_function: Optional[Callable[[], str]] = None,
383        stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None,
384        stop_event: Optional[asyncio.Event] = None,
385        stop_on_exit: bool = False,
386        strip_timestamps: bool = False,
387        accept_input: bool = True,
388        debug: bool = False,
389        _logs_path: Optional[pathlib.Path] = None,
390        _log=None,
391        _stdin_file=None,
392        _wait_if_stopped: bool = True,
393    ):
394        """
395        Monitor the job's log files and execute a callback on new lines.
396
397        Parameters
398        ----------
399        callback_function: Callable[[str], None], default partial(print, end='')
400            The callback to execute as new data comes in.
401            Defaults to printing the output directly to `stdout`.
402
403        input_callback_function: Optional[Callable[[], str]], default None
404            If provided, execute this callback when the daemon is blocking on stdin.
405            Defaults to `sys.stdin.readline()`.
406
407        stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None
408            If provided, execute this callback when the daemon stops.
409            The job's SuccessTuple will be passed to the callback.
410
411        stop_event: Optional[asyncio.Event], default None
412            If provided, stop monitoring when this event is set.
413            You may instead raise `meerschaum.jobs.StopMonitoringLogs`
414            from within `callback_function` to stop monitoring.
415
416        stop_on_exit: bool, default False
417            If `True`, stop monitoring when the job stops.
418
419        strip_timestamps: bool, default False
420            If `True`, remove leading timestamps from lines.
421
422        accept_input: bool, default True
423            If `True`, accept input when the daemon blocks on stdin.
424        """
425        if self.executor is not None:
426            self.executor.monitor_logs(
427                self.name,
428                callback_function,
429                input_callback_function=input_callback_function,
430                stop_callback_function=stop_callback_function,
431                stop_on_exit=stop_on_exit,
432                accept_input=accept_input,
433                strip_timestamps=strip_timestamps,
434                debug=debug,
435            )
436            return
437
438        monitor_logs_coroutine = self.monitor_logs_async(
439            callback_function=callback_function,
440            input_callback_function=input_callback_function,
441            stop_callback_function=stop_callback_function,
442            stop_event=stop_event,
443            stop_on_exit=stop_on_exit,
444            strip_timestamps=strip_timestamps,
445            accept_input=accept_input,
446            debug=debug,
447            _logs_path=_logs_path,
448            _log=_log,
449            _stdin_file=_stdin_file,
450            _wait_if_stopped=_wait_if_stopped,
451        )
452        return asyncio.run(monitor_logs_coroutine)

Monitor the job's log files and execute a callback on new lines.

Parameters
  • callback_function (Callable[[str], None], default partial(print, end='')): The callback to execute as new data comes in. Defaults to printing the output directly to stdout.
  • input_callback_function (Optional[Callable[[], str]], default None): If provided, execute this callback when the daemon is blocking on stdin. Defaults to sys.stdin.readline().
  • stop_callback_function (Optional[Callable[[SuccessTuple]], str], default None): If provided, execute this callback when the daemon stops. The job's SuccessTuple will be passed to the callback.
  • stop_event (Optional[asyncio.Event], default None): If provided, stop monitoring when this event is set. You may instead raise meerschaum.jobs.StopMonitoringLogs from within callback_function to stop monitoring.
  • stop_on_exit (bool, default False): If True, stop monitoring when the job stops.
  • strip_timestamps (bool, default False): If True, remove leading timestamps from lines.
  • accept_input (bool, default True): If True, accept input when the daemon blocks on stdin.
async def monitor_logs_async( self, callback_function: Callable[[str], NoneType] = <function _default_stdout_callback>, input_callback_function: Optional[Callable[[], str]] = None, stop_callback_function: Optional[Callable[[Tuple[bool, str]], NoneType]] = None, stop_event: Optional[asyncio.locks.Event] = None, stop_on_exit: bool = False, strip_timestamps: bool = False, accept_input: bool = True, debug: bool = False, _logs_path: Optional[pathlib.Path] = None, _log=None, _stdin_file=None, _wait_if_stopped: bool = True):
454    async def monitor_logs_async(
455        self,
456        callback_function: Callable[[str], None] = _default_stdout_callback,
457        input_callback_function: Optional[Callable[[], str]] = None,
458        stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None,
459        stop_event: Optional[asyncio.Event] = None,
460        stop_on_exit: bool = False,
461        strip_timestamps: bool = False,
462        accept_input: bool = True,
463        debug: bool = False,
464        _logs_path: Optional[pathlib.Path] = None,
465        _log=None,
466        _stdin_file=None,
467        _wait_if_stopped: bool = True,
468    ):
469        """
470        Monitor the job's log files and await a callback on new lines.
471
472        Parameters
473        ----------
474        callback_function: Callable[[str], None], default _default_stdout_callback
475            The callback to execute as new data comes in.
476            Defaults to printing the output directly to `stdout`.
477
478        input_callback_function: Optional[Callable[[], str]], default None
479            If provided, execute this callback when the daemon is blocking on stdin.
480            Defaults to `sys.stdin.readline()`.
481
482        stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None
483            If provided, execute this callback when the daemon stops.
484            The job's SuccessTuple will be passed to the callback.
485
486        stop_event: Optional[asyncio.Event], default None
487            If provided, stop monitoring when this event is set.
488            You may instead raise `meerschaum.jobs.StopMonitoringLogs`
489            from within `callback_function` to stop monitoring.
490
491        stop_on_exit: bool, default False
492            If `True`, stop monitoring when the job stops.
493
494        strip_timestamps: bool, default False
495            If `True`, remove leading timestamps from lines.
496
497        accept_input: bool, default True
498            If `True`, accept input when the daemon blocks on stdin.
499        """
500        from meerschaum.utils.prompt import prompt
501
502        def default_input_callback_function():
503            prompt_kwargs = self.get_prompt_kwargs(debug=debug)
504            if prompt_kwargs:
505                answer = prompt(**prompt_kwargs)
506                return answer + '\n'
507            return sys.stdin.readline()
508
509        if input_callback_function is None:
510            input_callback_function = default_input_callback_function
511
512        if self.executor is not None:
513            await self.executor.monitor_logs_async(
514                self.name,
515                callback_function,
516                input_callback_function=input_callback_function,
517                stop_callback_function=stop_callback_function,
518                stop_on_exit=stop_on_exit,
519                strip_timestamps=strip_timestamps,
520                accept_input=accept_input,
521                debug=debug,
522            )
523            return
524
525        from meerschaum.utils.formatting._jobs import strip_timestamp_from_line
526
527        events = {
528            'user': stop_event,
529            'stopped': asyncio.Event(),
530            'stop_token': asyncio.Event(),
531            'stop_exception': asyncio.Event(),
532            'stopped_timeout': asyncio.Event(),
533        }
534        combined_event = asyncio.Event()
535        emitted_text = False
536        stdin_file = _stdin_file if _stdin_file is not None else self.daemon.stdin_file
537
538        async def check_job_status():
539            if not stop_on_exit:
540                return
541
542            nonlocal emitted_text
543
544            sleep_time = 0.1
545            while sleep_time < 0.2:
546                if self.status == 'stopped':
547                    if not emitted_text and _wait_if_stopped:
548                        await asyncio.sleep(sleep_time)
549                        sleep_time = round(sleep_time * 1.1, 3)
550                        continue
551
552                    if stop_callback_function is not None:
553                        try:
554                            if asyncio.iscoroutinefunction(stop_callback_function):
555                                await stop_callback_function(self.result)
556                            else:
557                                stop_callback_function(self.result)
558                        except asyncio.exceptions.CancelledError:
559                            break
560                        except Exception:
561                            warn(traceback.format_exc())
562
563                    if stop_on_exit:
564                        events['stopped'].set()
565
566                    break
567                await asyncio.sleep(0.1)
568
569            events['stopped_timeout'].set()
570
571        async def check_blocking_on_input():
572            while True:
573                if not emitted_text or not self.is_blocking_on_stdin():
574                    try:
575                        await asyncio.sleep(self.refresh_seconds)
576                    except asyncio.exceptions.CancelledError:
577                        break
578                    continue
579
580                if not self.is_running():
581                    break
582
583                await emit_latest_lines()
584
585                try:
586                    print('', end='', flush=True)
587                    if asyncio.iscoroutinefunction(input_callback_function):
588                        data = await input_callback_function()
589                    else:
590                        loop = asyncio.get_running_loop()
591                        data = await loop.run_in_executor(None, input_callback_function)
592                except KeyboardInterrupt:
593                    break
594                #  if not data.endswith('\n'):
595                    #  data += '\n'
596
597                stdin_file.write(data)
598                await asyncio.sleep(self.refresh_seconds)
599
600        async def combine_events():
601            event_tasks = [
602                asyncio.create_task(event.wait())
603                for event in events.values()
604                if event is not None
605            ]
606            if not event_tasks:
607                return
608
609            try:
610                done, pending = await asyncio.wait(
611                    event_tasks,
612                    return_when=asyncio.FIRST_COMPLETED,
613                )
614                for task in pending:
615                    task.cancel()
616            except asyncio.exceptions.CancelledError:
617                pass
618            finally:
619                combined_event.set()
620
621        check_job_status_task = asyncio.create_task(check_job_status())
622        check_blocking_on_input_task = asyncio.create_task(check_blocking_on_input())
623        combine_events_task = asyncio.create_task(combine_events())
624
625        log = _log if _log is not None else self.daemon.rotating_log
626        lines_to_show = (
627            self.daemon.properties.get(
628                'logs', {}
629            ).get(
630                'lines_to_show', get_config('jobs', 'logs', 'lines_to_show')
631            )
632        )
633
634        async def emit_latest_lines():
635            nonlocal emitted_text
636            nonlocal stop_event
637            lines = log.readlines()
638            for line in lines[(-1 * lines_to_show):]:
639                if stop_event is not None and stop_event.is_set():
640                    return
641
642                line_stripped_extra = strip_timestamp_from_line(line.strip())
643                line_stripped = strip_timestamp_from_line(line)
644
645                if line_stripped_extra == STOP_TOKEN:
646                    events['stop_token'].set()
647                    return
648
649                if line_stripped_extra == CLEAR_TOKEN:
650                    clear_screen(debug=debug)
651                    continue
652
653                if line_stripped_extra == FLUSH_TOKEN.strip():
654                    line_stripped = ''
655                    line = ''
656
657                if strip_timestamps:
658                    line = line_stripped
659
660                try:
661                    if asyncio.iscoroutinefunction(callback_function):
662                        await callback_function(line)
663                    else:
664                        callback_function(line)
665                    emitted_text = True
666                except StopMonitoringLogs:
667                    events['stop_exception'].set()
668                    return
669                except Exception:
670                    warn(f"Error in logs callback:\n{traceback.format_exc()}")
671
672        await emit_latest_lines()
673
674        tasks = (
675            [check_job_status_task]
676            + ([check_blocking_on_input_task] if accept_input else [])
677            + [combine_events_task]
678        )
679        try:
680            _ = asyncio.gather(*tasks, return_exceptions=True)
681        except asyncio.exceptions.CancelledError:
682            raise
683        except Exception:
684            warn(f"Failed to run async checks:\n{traceback.format_exc()}")
685
686        watchfiles = mrsm.attempt_import('watchfiles', lazy=False)
687        dir_path_to_monitor = (
688            _logs_path
689            or (log.file_path.parent if log else None)
690            or paths.LOGS_RESOURCES_PATH
691        )
692        async for changes in watchfiles.awatch(
693            dir_path_to_monitor,
694            stop_event=combined_event,
695        ):
696            for change in changes:
697                file_path_str = change[1]
698                file_path = pathlib.Path(file_path_str)
699                latest_subfile_path = log.get_latest_subfile_path()
700                if latest_subfile_path != file_path:
701                    continue
702
703                await emit_latest_lines()
704
705        await emit_latest_lines()

Monitor the job's log files and await a callback on new lines.

Parameters
  • callback_function (Callable[[str], None], default _default_stdout_callback): The callback to execute as new data comes in. Defaults to printing the output directly to stdout.
  • input_callback_function (Optional[Callable[[], str]], default None): If provided, execute this callback when the daemon is blocking on stdin. Defaults to sys.stdin.readline().
  • stop_callback_function (Optional[Callable[[SuccessTuple]], str], default None): If provided, execute this callback when the daemon stops. The job's SuccessTuple will be passed to the callback.
  • stop_event (Optional[asyncio.Event], default None): If provided, stop monitoring when this event is set. You may instead raise meerschaum.jobs.StopMonitoringLogs from within callback_function to stop monitoring.
  • stop_on_exit (bool, default False): If True, stop monitoring when the job stops.
  • strip_timestamps (bool, default False): If True, remove leading timestamps from lines.
  • accept_input (bool, default True): If True, accept input when the daemon blocks on stdin.
def is_blocking_on_stdin(self, debug: bool = False) -> bool:
707    def is_blocking_on_stdin(self, debug: bool = False) -> bool:
708        """
709        Return whether a job's daemon is blocking on stdin.
710        """
711        if self.executor is not None:
712            return self.executor.get_job_is_blocking_on_stdin(self.name, debug=debug)
713
714        return self.is_running() and self.daemon.blocking_stdin_file_path.exists()

Return whether a job's daemon is blocking on stdin.

def get_prompt_kwargs(self, debug: bool = False) -> Dict[str, Any]:
716    def get_prompt_kwargs(self, debug: bool = False) -> Dict[str, Any]:
717        """
718        Return the kwargs to the blocking `prompt()`, if available.
719        """
720        if self.executor is not None:
721            return self.executor.get_job_prompt_kwargs(self.name, debug=debug)
722
723        if not self.daemon.prompt_kwargs_file_path.exists():
724            return {}
725
726        try:
727            with open(self.daemon.prompt_kwargs_file_path, 'r', encoding='utf-8') as f:
728                prompt_kwargs = json.load(f)
729
730            return prompt_kwargs
731        
732        except Exception:
733            import traceback
734            traceback.print_exc()
735            return {}

Return the kwargs to the blocking prompt(), if available.

def write_stdin(self, data):
737    def write_stdin(self, data):
738        """
739        Write to a job's daemon's `stdin`.
740        """
741        self.daemon.stdin_file.write(data)

Write to a job's daemon's stdin.

executor: Optional[meerschaum.jobs.Executor]
743    @property
744    def executor(self) -> Union[Executor, None]:
745        """
746        If the job is remote, return the connector to the remote API instance.
747        """
748        return (
749            mrsm.get_connector(self.executor_keys)
750            if self.executor_keys != 'local'
751            else None
752        )

If the job is remote, return the connector to the remote API instance.

status: str
754    @property
755    def status(self) -> str:
756        """
757        Return the running status of the job's daemon.
758        """
759        if '_status_hook' in self.__dict__:
760            return self._status_hook()
761
762        if self.executor is not None:
763            return self.executor.get_job_status(self.name)
764
765        return self.daemon.status

Return the running status of the job's daemon.

pid: Optional[int]
767    @property
768    def pid(self) -> Union[int, None]:
769        """
770        Return the PID of the job's dameon.
771        """
772        if self.executor is not None:
773            return self.executor.get_job_metadata(self.name).get('daemon', {}).get('pid', None)
774
775        return self.daemon.pid

Return the PID of the job's dameon.

restart: bool
777    @property
778    def restart(self) -> bool:
779        """
780        Return whether to restart a stopped job.
781        """
782        if self.executor is not None:
783            return self.executor.get_job_metadata(self.name).get('restart', False)
784
785        return self.daemon.properties.get('restart', False)

Return whether to restart a stopped job.

result: Tuple[bool, str]
787    @property
788    def result(self) -> SuccessTuple:
789        """
790        Return the `SuccessTuple` when the job has terminated.
791        """
792        if self.is_running():
793            return True, f"{self} is running."
794
795        if '_result_hook' in self.__dict__:
796            return self._result_hook()
797
798        if self.executor is not None:
799            return (
800                self.executor.get_job_metadata(self.name)
801                .get('result', (False, "No result available."))
802            )
803
804        _result = self.daemon.properties.get('result', None)
805        if _result is None:
806            from meerschaum.utils.daemon.Daemon import _results
807            return _results.get(self.daemon.daemon_id, (False, "No result available."))
808
809        return tuple(_result)

Return the SuccessTuple when the job has terminated.

sysargs: List[str]
811    @property
812    def sysargs(self) -> List[str]:
813        """
814        Return the sysargs to use for the Daemon.
815        """
816        if self._sysargs:
817            return self._sysargs
818
819        if self.executor is not None:
820            return self.executor.get_job_metadata(self.name).get('sysargs', [])
821
822        target_args = self.daemon.target_args
823        if target_args is None:
824            return []
825        self._sysargs = target_args[0] if len(target_args) > 0 else []
826        return self._sysargs

Return the sysargs to use for the Daemon.

def get_daemon_properties(self) -> Dict[str, Any]:
828    def get_daemon_properties(self) -> Dict[str, Any]:
829        """
830        Return the `properties` dictionary for the job's daemon.
831        """
832        remote_properties = (
833            {}
834            if self.executor is None
835            else self.executor.get_job_properties(self.name)
836        )
837        return {
838            **remote_properties,
839            **self._properties_patch
840        }

Return the properties dictionary for the job's daemon.

daemon: "'Daemon'"
842    @property
843    def daemon(self) -> 'Daemon':
844        """
845        Return the daemon which this job manages.
846        """
847        from meerschaum.utils.daemon import Daemon
848        if self._daemon is not None and self.executor is None and self._sysargs:
849            return self._daemon
850
851        self._daemon = Daemon(
852            target=entry,
853            target_args=[self._sysargs],
854            target_kw={},
855            daemon_id=self.name,
856            label=shlex.join(self._sysargs),
857            properties=self.get_daemon_properties(),
858        )
859        if '_rotating_log' in self.__dict__:
860            self._daemon._rotating_log = self._rotating_log
861
862        if '_stdin_file' in self.__dict__:
863            self._daemon._stdin_file = self._stdin_file
864            self._daemon._blocking_stdin_file_path = self._stdin_file.blocking_file_path
865
866        return self._daemon

Return the daemon which this job manages.

began: Optional[datetime.datetime]
868    @property
869    def began(self) -> Union[datetime, None]:
870        """
871        The datetime when the job began running.
872        """
873        if self.executor is not None:
874            began_str = self.executor.get_job_began(self.name)
875            if began_str is None:
876                return None
877            return (
878                datetime.fromisoformat(began_str)
879                .astimezone(timezone.utc)
880                .replace(tzinfo=None)
881            )
882
883        began_str = self.daemon.properties.get('process', {}).get('began', None)
884        if began_str is None:
885            return None
886
887        return datetime.fromisoformat(began_str)

The datetime when the job began running.

ended: Optional[datetime.datetime]
889    @property
890    def ended(self) -> Union[datetime, None]:
891        """
892        The datetime when the job stopped running.
893        """
894        if self.executor is not None:
895            ended_str = self.executor.get_job_ended(self.name)
896            if ended_str is None:
897                return None
898            return (
899                datetime.fromisoformat(ended_str)
900                .astimezone(timezone.utc)
901                .replace(tzinfo=None)
902            )
903
904        ended_str = self.daemon.properties.get('process', {}).get('ended', None)
905        if ended_str is None:
906            return None
907
908        return datetime.fromisoformat(ended_str)

The datetime when the job stopped running.

paused: Optional[datetime.datetime]
910    @property
911    def paused(self) -> Union[datetime, None]:
912        """
913        The datetime when the job was suspended while running.
914        """
915        if self.executor is not None:
916            paused_str = self.executor.get_job_paused(self.name)
917            if paused_str is None:
918                return None
919            return (
920                datetime.fromisoformat(paused_str)
921                .astimezone(timezone.utc)
922                .replace(tzinfo=None)
923            )
924
925        paused_str = self.daemon.properties.get('process', {}).get('paused', None)
926        if paused_str is None:
927            return None
928
929        return datetime.fromisoformat(paused_str)

The datetime when the job was suspended while running.

stop_time: Optional[datetime.datetime]
931    @property
932    def stop_time(self) -> Union[datetime, None]:
933        """
934        Return the timestamp when the job was manually stopped.
935        """
936        if self.executor is not None:
937            return self.executor.get_job_stop_time(self.name)
938
939        if not self.daemon.stop_path.exists():
940            return None
941
942        stop_data = self.daemon._read_stop_file()
943        if not stop_data:
944            return None
945
946        stop_time_str = stop_data.get('stop_time', None)
947        if not stop_time_str:
948            warn(f"Could not read stop time for {self}.")
949            return None
950
951        return datetime.fromisoformat(stop_time_str)

Return the timestamp when the job was manually stopped.

hidden: bool
953    @property
954    def hidden(self) -> bool:
955        """
956        Return a bool indicating whether this job should be displayed.
957        """
958        return (
959            self.name.startswith('_')
960            or self.name.startswith('.')
961            or self._is_externally_managed
962        )

Return a bool indicating whether this job should be displayed.

def check_restart(self) -> Tuple[bool, str]:
964    def check_restart(self) -> SuccessTuple:
965        """
966        If `restart` is `True` and the daemon is not running,
967        restart the job.
968        Do not restart if the job was manually stopped.
969        """
970        if self.is_running():
971            return True, f"{self} is running."
972
973        if not self.restart:
974            return True, f"{self} does not need to be restarted."
975
976        if self.stop_time is not None:
977            return True, f"{self} was manually stopped."
978
979        return self.start()

If restart is True and the daemon is not running, restart the job. Do not restart if the job was manually stopped.

label: str
981    @property
982    def label(self) -> str:
983        """
984        Return the job's Daemon label (joined sysargs).
985        """
986        from meerschaum._internal.arguments import compress_pipeline_sysargs
987        sysargs = compress_pipeline_sysargs(self.sysargs)
988        return shlex.join(sysargs).replace(' + ', '\n+ ').replace(' : ', '\n: ').lstrip().rstrip()

Return the job's Daemon label (joined sysargs).

env: Dict[str, str]
1017    @property
1018    def env(self) -> Dict[str, str]:
1019        """
1020        Return the environment variables to set for the job's process.
1021        """
1022        if '_env' in self.__dict__:
1023            return self.__dict__['_env']
1024
1025        _env = self.daemon.properties.get('env', {})
1026        default_env = {
1027            'PYTHONUNBUFFERED': '1',
1028            'LINES': str(get_config('jobs', 'terminal', 'lines')),
1029            'COLUMNS': str(get_config('jobs', 'terminal', 'columns')),
1030            STATIC_CONFIG['environment']['noninteractive']: 'true',
1031        }
1032        self._env = {**default_env, **_env}
1033        return self._env

Return the environment variables to set for the job's process.

delete_after_completion: bool
1035    @property
1036    def delete_after_completion(self) -> bool:
1037        """
1038        Return whether this job is configured to delete itself after completion.
1039        """
1040        if '_delete_after_completion' in self.__dict__:
1041            return self.__dict__.get('_delete_after_completion', False)
1042
1043        self._delete_after_completion = self.daemon.properties.get('delete_after_completion', False)
1044        return self._delete_after_completion

Return whether this job is configured to delete itself after completion.

def pprint( *args, detect_password: bool = True, nopretty: bool = False, **kw) -> None:
 10def pprint(
 11    *args,
 12    detect_password: bool = True,
 13    nopretty: bool = False,
 14    **kw
 15) -> None:
 16    """Pretty print an object according to the configured ANSI and UNICODE settings.
 17    If detect_password is True (default), search and replace passwords with '*' characters.
 18    Does not mutate objects.
 19    """
 20    import copy
 21    import json
 22    from meerschaum.utils.packages import attempt_import, import_rich
 23    from meerschaum.utils.formatting import ANSI, get_console, print_tuple
 24    from meerschaum.utils.warnings import error
 25    from meerschaum.utils.misc import replace_password, dict_from_od, filter_keywords
 26    from collections import OrderedDict
 27
 28    if (
 29        len(args) == 1
 30        and
 31        isinstance(args[0], tuple)
 32        and
 33        len(args[0]) == 2
 34        and
 35        isinstance(args[0][0], bool)
 36        and
 37        isinstance(args[0][1], str)
 38    ):
 39        return print_tuple(args[0], **filter_keywords(print_tuple, **kw))
 40
 41    modify = True
 42    rich_pprint = None
 43    if ANSI and not nopretty:
 44        rich = import_rich()
 45        if rich is not None:
 46            rich_pretty = attempt_import('rich.pretty')
 47        if rich_pretty is not None:
 48            def _rich_pprint(*args, **kw):
 49                _console = get_console()
 50                _kw = filter_keywords(_console.print, **kw)
 51                _console.print(*args, **_kw)
 52            rich_pprint = _rich_pprint
 53    elif not nopretty:
 54        pprintpp = attempt_import('pprintpp', warn=False)
 55        try:
 56            _pprint = pprintpp.pprint
 57        except Exception :
 58            import pprint as _pprint_module
 59            _pprint = _pprint_module.pprint
 60
 61    func = (
 62        _pprint if rich_pprint is None else rich_pprint
 63    ) if not nopretty else print
 64
 65    try:
 66        args_copy = copy.deepcopy(args)
 67    except Exception:
 68        args_copy = args
 69        modify = False
 70
 71    _args = []
 72    for a in args:
 73        c = a
 74        ### convert OrderedDict into dict
 75        if isinstance(a, OrderedDict) or issubclass(type(a), OrderedDict):
 76            c = dict_from_od(copy.deepcopy(c))
 77        _args.append(c)
 78    args = _args
 79
 80    _args = list(args)
 81    if detect_password and modify:
 82        _args = []
 83        for a in args:
 84            c = a
 85            if isinstance(c, dict):
 86                c = replace_password(copy.deepcopy(c))
 87            if nopretty:
 88                try:
 89                    c = json.dumps(c)
 90                    is_json = True
 91                except Exception:
 92                    is_json = False
 93                if not is_json:
 94                    try:
 95                        c = str(c)
 96                    except Exception:
 97                        pass
 98            _args.append(c)
 99
100    ### filter out unsupported keywords
101    func_kw = filter_keywords(func, **kw) if not nopretty else {}
102    error_msg = None
103    try:
104        func(*_args, **func_kw)
105    except Exception as e:
106        error_msg = e
107    if error_msg is not None:
108        error(error_msg)

Pretty print an object according to the configured ANSI and UNICODE settings. If detect_password is True (default), search and replace passwords with '*' characters. Does not mutate objects.

def attempt_import( *names: str, lazy: bool = True, warn: bool = True, install: bool = True, venv: Optional[str] = 'mrsm', precheck: bool = True, split: bool = True, check_update: bool = False, check_pypi: bool = False, check_is_installed: bool = True, allow_outside_venv: bool = True, color: bool = True, debug: bool = False) -> Any:
1418def attempt_import(
1419    *names: str,
1420    lazy: bool = True,
1421    warn: bool = True,
1422    install: bool = True,
1423    venv: Optional[str] = 'mrsm',
1424    precheck: bool = True,
1425    split: bool = True,
1426    check_update: bool = False,
1427    check_pypi: bool = False,
1428    check_is_installed: bool = True,
1429    allow_outside_venv: bool = True,
1430    color: bool = True,
1431    debug: bool = False
1432) -> Any:
1433    """
1434    Raise a warning if packages are not installed; otherwise import and return modules.
1435    If `lazy` is `True`, return lazy-imported modules.
1436    
1437    Returns tuple of modules if multiple names are provided, else returns one module.
1438    
1439    Parameters
1440    ----------
1441    names: List[str]
1442        The packages to be imported.
1443
1444    lazy: bool, default True
1445        If `True`, lazily load packages.
1446
1447    warn: bool, default True
1448        If `True`, raise a warning if a package cannot be imported.
1449
1450    install: bool, default True
1451        If `True`, attempt to install a missing package into the designated virtual environment.
1452        If `check_update` is True, install updates if available.
1453
1454    venv: Optional[str], default 'mrsm'
1455        The virtual environment in which to search for packages and to install packages into.
1456
1457    precheck: bool, default True
1458        If `True`, attempt to find module before importing (necessary for checking if modules exist
1459        and retaining lazy imports), otherwise assume lazy is `False`.
1460
1461    split: bool, default True
1462        If `True`, split packages' names on `'.'`.
1463
1464    check_update: bool, default False
1465        If `True` and `install` is `True`, install updates if the required minimum version
1466        does not match.
1467
1468    check_pypi: bool, default False
1469        If `True` and `check_update` is `True`, check PyPI when determining whether
1470        an update is required.
1471
1472    check_is_installed: bool, default True
1473        If `True`, check if the package is contained in the virtual environment.
1474
1475    allow_outside_venv: bool, default True
1476        If `True`, search outside of the specified virtual environment
1477        if the package cannot be found.
1478        Setting to `False` will reinstall the package into a virtual environment, even if it
1479        is installed outside.
1480
1481    color: bool, default True
1482        If `False`, do not print ANSI colors.
1483
1484    Returns
1485    -------
1486    The specified modules. If they're not available and `install` is `True`, it will first
1487    download them into a virtual environment and return the modules.
1488
1489    Examples
1490    --------
1491    >>> pandas, sqlalchemy = attempt_import('pandas', 'sqlalchemy')
1492    >>> pandas = attempt_import('pandas')
1493
1494    """
1495
1496    import importlib.util
1497
1498    global emitted_auto_install_warning
1499    no_auto_install_env_var = 'MRSM_NO_AUTO_INSTALL'
1500    if os.environ.get(no_auto_install_env_var, '').lower() in ('1', 'true', 'yes'):
1501        install = False
1502
1503    ### to prevent recursion, check if parent Meerschaum package is being imported
1504    if names == ('meerschaum',):
1505        return _import_module('meerschaum')
1506
1507    if venv == 'mrsm' and _import_hook_venv is not None:
1508        if debug:
1509            print(f"Import hook for virtual environment '{_import_hook_venv}' is active.")
1510        venv = _import_hook_venv
1511
1512    _warnings = _import_module('meerschaum.utils.warnings')
1513    warn_function = _warnings.warn
1514
1515    def do_import(_name: str, **kw) -> Union['ModuleType', None]:
1516        ### A fully-imported module resolves from `sys.modules` regardless of the active venv,
1517        ### so skip activating the venv (which resolves and creates its directories).
1518        import sys
1519        loaded_module = sys.modules.get(_name, None)
1520        if (
1521            loaded_module is not None
1522            and not getattr(getattr(loaded_module, '__spec__', None), '_initializing', False)
1523        ):
1524            return loaded_module
1525
1526        with Venv(venv=venv, debug=debug):
1527            ### determine the import method (lazy vs normal)
1528            from meerschaum.utils.misc import filter_keywords
1529            import_method = (
1530                _import_module if not lazy
1531                else lazy_import
1532            )
1533            try:
1534                mod = import_method(_name, **(filter_keywords(import_method, **kw)))
1535            except Exception as e:
1536                if warn:
1537                    import traceback
1538                    traceback.print_exception(type(e), e, e.__traceback__)
1539                    warn_function(
1540                        f"Failed to import module '{_name}'.\nException:\n{e}",
1541                        ImportWarning,
1542                        stacklevel = (5 if lazy else 4),
1543                        color = False,
1544                    )
1545                mod = None
1546        return mod
1547
1548    modules = []
1549    for name in names:
1550        ### Check if package is a declared dependency.
1551        root_name = name.split('.')[0] if split else name
1552        install_name = _import_to_install_name(root_name)
1553
1554        if install_name is None:
1555            install_name = root_name
1556            if warn and root_name != 'plugins':
1557                warn_function(
1558                    f"Package '{root_name}' is not declared in meerschaum.utils.packages.",
1559                    ImportWarning,
1560                    stacklevel = 3,
1561                    color = False
1562                )
1563
1564        ### Determine if the package exists.
1565        if precheck is False:
1566            found_module = (
1567                do_import(
1568                    name, debug=debug, warn=False, venv=venv, color=color,
1569                    check_update=False, check_pypi=False, split=split,
1570                ) is not None
1571            )
1572        else:
1573            installed_cache_key = (name, venv, split, allow_outside_venv)
1574            if check_is_installed:
1575                with _locks['_is_installed_first_check']:
1576                    if not _is_installed_first_check.get(installed_cache_key, False):
1577                        package_is_installed = is_installed(
1578                            name,
1579                            venv = venv,
1580                            split = split,
1581                            allow_outside_venv = allow_outside_venv,
1582                            debug = debug,
1583                        )
1584                        if package_is_installed:
1585                            _is_installed_first_check[installed_cache_key] = True
1586                    else:
1587                        package_is_installed = True
1588            else:
1589                package_is_installed = venv_contains_package(
1590                    name,
1591                    venv=venv,
1592                    split=split,
1593                    debug=debug,
1594                )
1595            found_module = package_is_installed
1596
1597        if not found_module:
1598            if install:
1599                with _locks['emitted_auto_install_warning']:
1600                    if warn and not emitted_auto_install_warning:
1601                        emitted_auto_install_warning = True
1602                        warn_function(
1603                            "Meerschaum is installing a missing runtime dependency. "
1604                            + f"Set {no_auto_install_env_var}=1 to disable automatic downloads.",
1605                            ImportWarning,
1606                            stacklevel=3,
1607                            color=False,
1608                        )
1609                if not pip_install(
1610                    install_name,
1611                    venv = venv,
1612                    split = False,
1613                    check_update = check_update,
1614                    color = color,
1615                    debug = debug
1616                ) and warn:
1617                    warn_function(
1618                        f"Failed to install '{install_name}'.",
1619                        ImportWarning,
1620                        stacklevel = 3,
1621                        color = False,
1622                    )
1623            elif warn:
1624                ### Raise a warning if we can't find the package and install = False.
1625                warn_function(
1626                    (f"\n\nMissing package '{name}' from virtual environment '{venv}'; "
1627                     + "some features will not work correctly."
1628                     + (
1629                         f"\n\nUnset {no_auto_install_env_var} to allow package installation.\n"
1630                         if no_auto_install_env_var in os.environ
1631                         else "\n\nSet install=True when calling attempt_import.\n"
1632                     )),
1633                    ImportWarning,
1634                    stacklevel = 3,
1635                    color = False,
1636                )
1637
1638        ### Do the import. Will be lazy if lazy=True.
1639        m = do_import(
1640            name, debug=debug, warn=warn, venv=venv, color=color,
1641            check_update=check_update, check_pypi=check_pypi, install=install, split=split,
1642        )
1643        modules.append(m)
1644
1645    modules = tuple(modules)
1646    if len(modules) == 1:
1647        return modules[0]
1648    return modules

Raise a warning if packages are not installed; otherwise import and return modules. If lazy is True, return lazy-imported modules.

Returns tuple of modules if multiple names are provided, else returns one module.

Parameters
  • names (List[str]): The packages to be imported.
  • lazy (bool, default True): If True, lazily load packages.
  • warn (bool, default True): If True, raise a warning if a package cannot be imported.
  • install (bool, default True): If True, attempt to install a missing package into the designated virtual environment. If check_update is True, install updates if available.
  • venv (Optional[str], default 'mrsm'): The virtual environment in which to search for packages and to install packages into.
  • precheck (bool, default True): If True, attempt to find module before importing (necessary for checking if modules exist and retaining lazy imports), otherwise assume lazy is False.
  • split (bool, default True): If True, split packages' names on '.'.
  • check_update (bool, default False): If True and install is True, install updates if the required minimum version does not match.
  • check_pypi (bool, default False): If True and check_update is True, check PyPI when determining whether an update is required.
  • check_is_installed (bool, default True): If True, check if the package is contained in the virtual environment.
  • allow_outside_venv (bool, default True): If True, search outside of the specified virtual environment if the package cannot be found. Setting to False will reinstall the package into a virtual environment, even if it is installed outside.
  • color (bool, default True): If False, do not print ANSI colors.
Returns
  • The specified modules. If they're not available and install is True, it will first
  • download them into a virtual environment and return the modules.
Examples
>>> pandas, sqlalchemy = attempt_import('pandas', 'sqlalchemy')
>>> pandas = attempt_import('pandas')
class Connector:
 22class Connector(metaclass=abc.ABCMeta):
 23    """
 24    The base connector class to hold connection attributes.
 25    """
 26
 27    IS_INSTANCE: bool = False
 28
 29    def __init__(
 30        self,
 31        type: Optional[str] = None,
 32        label: Optional[str] = None,
 33        **kw: Any
 34    ):
 35        """
 36        Set the given keyword arguments as attributes.
 37
 38        Parameters
 39        ----------
 40        type: str
 41            The `type` of the connector (e.g. `sql`, `api`, `plugin`).
 42
 43        label: str
 44            The `label` for the connector.
 45
 46
 47        Examples
 48        --------
 49        Run `mrsm edit config` and to edit connectors in the YAML file:
 50
 51        ```yaml
 52        meerschaum:
 53            connections:
 54                {type}:
 55                    {label}:
 56                        ### attributes go here
 57        ```
 58
 59        """
 60        self._original_dict = copy.deepcopy(self.__dict__)
 61        self._set_attributes(type=type, label=label, **kw)
 62
 63        ### NOTE: Override `REQUIRED_ATTRIBUTES` if `uri` is set.
 64        self.verify_attributes(
 65            ['uri']
 66            if 'uri' in self.__dict__
 67            else getattr(self, 'REQUIRED_ATTRIBUTES', None)
 68        )
 69
 70    def _reset_attributes(self):
 71        self.__dict__ = self._original_dict
 72
 73    def _set_attributes(
 74        self,
 75        *args,
 76        inherit_default: bool = True,
 77        **kw: Any
 78    ):
 79        from meerschaum._internal.static import STATIC_CONFIG
 80        from meerschaum.utils.warnings import error
 81
 82        self._attributes = {}
 83
 84        default_label = STATIC_CONFIG['connectors']['default_label']
 85
 86        ### NOTE: Support the legacy method of explicitly passing the type.
 87        label = kw.get('label', None)
 88        if label is None:
 89            if len(args) == 2:
 90                label = args[1]
 91            elif len(args) == 0:
 92                label = None
 93            else:
 94                label = args[0]
 95
 96        if label == 'default':
 97            error(
 98                f"Label cannot be 'default'. Did you mean '{default_label}'?",
 99                InvalidAttributesError,
100            )
101        self.__dict__['label'] = label
102
103        from meerschaum.config import get_config
104        conn_configs = copy.deepcopy(get_config('meerschaum', 'connectors'))
105        connector_config = copy.deepcopy(get_config('system', 'connectors'))
106
107        ### inherit attributes from 'default' if exists
108        if inherit_default:
109            inherit_from = 'default'
110            if self.type in conn_configs and inherit_from in conn_configs[self.type]:
111                _inherit_dict = copy.deepcopy(conn_configs[self.type][inherit_from])
112                self._attributes.update(_inherit_dict)
113
114        ### load user config into self._attributes
115        if self.type in conn_configs and self.label in conn_configs[self.type]:
116            self._attributes.update(conn_configs[self.type][self.label] or {})
117
118        ### load system config into self._sys_config
119        ### (deep copy so future Connectors don't inherit changes)
120        if self.type in connector_config:
121            self._sys_config = copy.deepcopy(connector_config[self.type])
122
123        ### add additional arguments or override configuration
124        self._attributes.update(kw)
125
126        ### finally, update __dict__ with _attributes.
127        self.__dict__.update(self._attributes)
128
129    def verify_attributes(
130        self,
131        required_attributes: Optional[List[str]] = None,
132        debug: bool = False,
133    ) -> None:
134        """
135        Ensure that the required attributes have been met.
136        
137        The Connector base class checks the minimum requirements.
138        Child classes may enforce additional requirements.
139
140        Parameters
141        ----------
142        required_attributes: Optional[List[str]], default None
143            Attributes to be verified. If `None`, default to `['label']`.
144
145        debug: bool, default False
146            Verbosity toggle.
147
148        Returns
149        -------
150        Don't return anything.
151
152        Raises
153        ------
154        An error if any of the required attributes are missing.
155        """
156        from meerschaum.utils.warnings import error
157        from meerschaum.utils.misc import items_str
158        if required_attributes is None:
159            required_attributes = ['type', 'label']
160
161        missing_attributes = set()
162        for a in required_attributes:
163            if a not in self.__dict__:
164                missing_attributes.add(a)
165        if len(missing_attributes) > 0:
166            error(
167                (
168                    f"Missing {items_str(list(missing_attributes))} "
169                    + f"for connector '{self.type}:{self.label}'."
170                ),
171                InvalidAttributesError,
172                silent=True,
173                stack=False
174            )
175
176
177    def __str__(self):
178        """
179        When cast to a string, return type:label.
180        """
181        return f"{self.type}:{self.label}"
182
183    def __repr__(self):
184        """
185        Represent the connector as type:label.
186        """
187        return str(self)
188
189    @property
190    def meta(self) -> Dict[str, Any]:
191        """
192        Return the keys needed to reconstruct this Connector.
193        """
194        _meta = {
195            key: value
196            for key, value in self.__dict__.items()
197            if not str(key).startswith('_')
198        }
199        _meta.update({
200            'type': self.type,
201            'label': self.label,
202        })
203        return _meta
204
205
206    @property
207    def type(self) -> str:
208        """
209        Return the type for this connector.
210        """
211        _type = self.__dict__.get('type', None)
212        if _type is None:
213            import re
214            is_executor = self.__class__.__name__.lower().endswith('executor')
215            suffix_regex = (
216                r'connector$'
217                if not is_executor
218                else r'executor$'
219            )
220            _type = re.sub(suffix_regex, '', self.__class__.__name__.lower())
221            if not _type or _type.lower() == 'instance':
222                raise ValueError("No type could be determined for this connector.")
223            self.__dict__['type'] = _type
224        return _type
225
226
227    @property
228    def label(self) -> str:
229        """
230        Return the label for this connector.
231        """
232        _label = self.__dict__.get('label', None)
233        if _label is None:
234            from meerschaum._internal.static import STATIC_CONFIG
235            _label = STATIC_CONFIG['connectors']['default_label']
236            self.__dict__['label'] = _label
237        return _label

The base connector class to hold connection attributes.

Connector(type: Optional[str] = None, label: Optional[str] = None, **kw: Any)
29    def __init__(
30        self,
31        type: Optional[str] = None,
32        label: Optional[str] = None,
33        **kw: Any
34    ):
35        """
36        Set the given keyword arguments as attributes.
37
38        Parameters
39        ----------
40        type: str
41            The `type` of the connector (e.g. `sql`, `api`, `plugin`).
42
43        label: str
44            The `label` for the connector.
45
46
47        Examples
48        --------
49        Run `mrsm edit config` and to edit connectors in the YAML file:
50
51        ```yaml
52        meerschaum:
53            connections:
54                {type}:
55                    {label}:
56                        ### attributes go here
57        ```
58
59        """
60        self._original_dict = copy.deepcopy(self.__dict__)
61        self._set_attributes(type=type, label=label, **kw)
62
63        ### NOTE: Override `REQUIRED_ATTRIBUTES` if `uri` is set.
64        self.verify_attributes(
65            ['uri']
66            if 'uri' in self.__dict__
67            else getattr(self, 'REQUIRED_ATTRIBUTES', None)
68        )

Set the given keyword arguments as attributes.

Parameters
  • type (str): The type of the connector (e.g. sql, api, plugin).
  • label (str): The label for the connector.
Examples

Run mrsm edit config and to edit connectors in the YAML file:

meerschaum:
    connections:
        {type}:
            {label}:
                ### attributes go here
IS_INSTANCE: bool = False
def verify_attributes( self, required_attributes: Optional[List[str]] = None, debug: bool = False) -> None:
129    def verify_attributes(
130        self,
131        required_attributes: Optional[List[str]] = None,
132        debug: bool = False,
133    ) -> None:
134        """
135        Ensure that the required attributes have been met.
136        
137        The Connector base class checks the minimum requirements.
138        Child classes may enforce additional requirements.
139
140        Parameters
141        ----------
142        required_attributes: Optional[List[str]], default None
143            Attributes to be verified. If `None`, default to `['label']`.
144
145        debug: bool, default False
146            Verbosity toggle.
147
148        Returns
149        -------
150        Don't return anything.
151
152        Raises
153        ------
154        An error if any of the required attributes are missing.
155        """
156        from meerschaum.utils.warnings import error
157        from meerschaum.utils.misc import items_str
158        if required_attributes is None:
159            required_attributes = ['type', 'label']
160
161        missing_attributes = set()
162        for a in required_attributes:
163            if a not in self.__dict__:
164                missing_attributes.add(a)
165        if len(missing_attributes) > 0:
166            error(
167                (
168                    f"Missing {items_str(list(missing_attributes))} "
169                    + f"for connector '{self.type}:{self.label}'."
170                ),
171                InvalidAttributesError,
172                silent=True,
173                stack=False
174            )

Ensure that the required attributes have been met.

The Connector base class checks the minimum requirements. Child classes may enforce additional requirements.

Parameters
  • required_attributes (Optional[List[str]], default None): Attributes to be verified. If None, default to ['label'].
  • debug (bool, default False): Verbosity toggle.
Returns
  • Don't return anything.
Raises
  • An error if any of the required attributes are missing.
meta: Dict[str, Any]
189    @property
190    def meta(self) -> Dict[str, Any]:
191        """
192        Return the keys needed to reconstruct this Connector.
193        """
194        _meta = {
195            key: value
196            for key, value in self.__dict__.items()
197            if not str(key).startswith('_')
198        }
199        _meta.update({
200            'type': self.type,
201            'label': self.label,
202        })
203        return _meta

Return the keys needed to reconstruct this Connector.

type: str
206    @property
207    def type(self) -> str:
208        """
209        Return the type for this connector.
210        """
211        _type = self.__dict__.get('type', None)
212        if _type is None:
213            import re
214            is_executor = self.__class__.__name__.lower().endswith('executor')
215            suffix_regex = (
216                r'connector$'
217                if not is_executor
218                else r'executor$'
219            )
220            _type = re.sub(suffix_regex, '', self.__class__.__name__.lower())
221            if not _type or _type.lower() == 'instance':
222                raise ValueError("No type could be determined for this connector.")
223            self.__dict__['type'] = _type
224        return _type

Return the type for this connector.

label: str
227    @property
228    def label(self) -> str:
229        """
230        Return the label for this connector.
231        """
232        _label = self.__dict__.get('label', None)
233        if _label is None:
234            from meerschaum._internal.static import STATIC_CONFIG
235            _label = STATIC_CONFIG['connectors']['default_label']
236            self.__dict__['label'] = _label
237        return _label

Return the label for this connector.

class InstanceConnector(meerschaum.Connector):
18class InstanceConnector(Connector):
19    """
20    Instance connectors define the interface for managing pipes and provide methods
21    for management of users, plugins, tokens, and other metadata built atop pipes.
22    """
23
24    IS_INSTANCE: bool = True
25    IS_THREAD_SAFE: bool = False
26
27    from ._users import (
28        get_users_pipe,
29        register_user,
30        get_user_id,
31        get_username,
32        get_users,
33        edit_user,
34        delete_user,
35        get_user_password_hash,
36        get_user_type,
37        get_user_attributes,
38    )
39
40    from ._plugins import (
41        get_plugins_pipe,
42        register_plugin,
43        get_plugin_user_id,
44        delete_plugin,
45        get_plugin_id,
46        get_plugin_version,
47        get_plugins,
48        get_plugin_user_id,
49        get_plugin_username,
50        get_plugin_attributes,
51    )
52
53    from ._tokens import (
54        get_tokens_pipe,
55        register_token,
56        edit_token,
57        invalidate_token,
58        delete_token,
59        get_token,
60        get_tokens,
61        get_token_model,
62        get_token_secret_hash,
63        token_exists,
64        get_token_scopes,
65    )
66
67    from ._pipes import (
68        register_pipe,
69        get_pipe_attributes,
70        get_pipe_id,
71        edit_pipe,
72        delete_pipe,
73        fetch_pipes_keys,
74        pipe_exists,
75        drop_pipe,
76        drop_pipe_indices,
77        sync_pipe,
78        create_pipe_indices,
79        clear_pipe,
80        get_pipe_data,
81        get_pipe_docs,
82        get_sync_time,
83        get_pipe_columns_types,
84        get_pipe_columns_indices,
85        get_pipe_size,
86        compress_pipe,
87        decompress_pipe,
88        vacuum_pipe,
89        analyze_pipe,
90        partition_pipe,
91    )

Instance connectors define the interface for managing pipes and provide methods for management of users, plugins, tokens, and other metadata built atop pipes.

IS_INSTANCE: bool = True
IS_THREAD_SAFE: bool = False
def get_users_pipe(self) -> Pipe:
18def get_users_pipe(self) -> 'mrsm.Pipe':
19    """
20    Return the pipe used for users registration.
21    """
22    if '_users_pipe' in self.__dict__:
23        return self._users_pipe
24
25    cache_connector = self.__dict__.get('_cache_connector', None)
26    self._users_pipe = mrsm.Pipe(
27        'mrsm', 'users',
28        instance=self,
29        target='mrsm_users',
30        temporary=True,
31        cache=True,
32        cache_connector_keys=cache_connector,
33        static=True,
34        null_indices=False,
35        columns={
36            'primary': 'user_id',
37        },
38        dtypes={
39            'user_id': 'uuid',
40            'username': 'string',
41            'password_hash': 'string',
42            'email': 'string',
43            'user_type': 'string',
44            'attributes': 'json',
45        },
46        indices={
47            'unique': 'username',
48        },
49    )
50    return self._users_pipe

Return the pipe used for users registration.

def register_user( self, user: meerschaum.core.User._User.User, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
53def register_user(
54    self,
55    user: User,
56    debug: bool = False,
57    **kwargs: Any
58) -> mrsm.SuccessTuple:
59    """
60    Register a new user to the users pipe.
61    """
62    users_pipe = self.get_users_pipe()
63    user.user_id = uuid.uuid4()
64    sync_success, sync_msg = users_pipe.sync(
65        [{
66            'user_id': user.user_id,
67            'username': user.username,
68            'email': user.email,
69            'password_hash': user.password_hash,
70            'user_type': user.type,
71            'attributes': user.attributes,
72        }],
73        check_existing=False,
74        debug=debug,
75    )
76    if not sync_success:
77        return False, f"Failed to register user '{user.username}':\n{sync_msg}"
78
79    return True, "Success"

Register a new user to the users pipe.

def get_user_id( self, user: meerschaum.core.User._User.User, debug: bool = False) -> Optional[uuid.UUID]:
82def get_user_id(self, user: User, debug: bool = False) -> Union[uuid.UUID, None]:
83    """
84    Return a user's ID from the username.
85    """
86    users_pipe = self.get_users_pipe()
87    result_df = users_pipe.get_data(['user_id'], params={'username': user.username}, limit=1)
88    if result_df is None or len(result_df) == 0:
89        return None
90    return result_df['user_id'][0]

Return a user's ID from the username.

def get_username(self, user_id: Any, debug: bool = False) -> Any:
93def get_username(self, user_id: Any, debug: bool = False) -> Any:
94    """
95    Return the username from the given ID.
96    """
97    users_pipe = self.get_users_pipe()
98    return users_pipe.get_value('username', {'user_id': user_id}, debug=debug)

Return the username from the given ID.

def get_users(self, debug: bool = False, **kw: Any) -> List[str]:
101def get_users(
102    self,
103    debug: bool = False,
104    **kw: Any
105) -> List[str]:
106    """
107    Get the registered usernames.
108    """
109    users_pipe = self.get_users_pipe()
110    df = users_pipe.get_data()
111    if df is None:
112        return []
113
114    return list(df['username'])

Get the registered usernames.

def edit_user( self, user: meerschaum.core.User._User.User, debug: bool = False) -> Tuple[bool, str]:
117def edit_user(self, user: User, debug: bool = False) -> mrsm.SuccessTuple:
118    """
119    Edit the attributes for an existing user.
120    """
121    users_pipe = self.get_users_pipe()
122    user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug)
123
124    doc = {'user_id': user_id}
125    if user.email != '':
126        doc['email'] = user.email
127    if user.password_hash != '':
128        doc['password_hash'] = user.password_hash
129    if user.type != '':
130        doc['user_type'] = user.type
131    if user.attributes:
132        doc['attributes'] = user.attributes
133
134    sync_success, sync_msg = users_pipe.sync([doc], debug=debug)
135    if not sync_success:
136        return False, f"Failed to edit user '{user.username}':\n{sync_msg}"
137
138    return True, "Success"

Edit the attributes for an existing user.

def delete_user( self, user: meerschaum.core.User._User.User, debug: bool = False) -> Tuple[bool, str]:
141def delete_user(self, user: User, debug: bool = False) -> mrsm.SuccessTuple:
142    """
143    Delete a user from the users table.
144    """
145    user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug)
146    users_pipe = self.get_users_pipe()
147    clear_success, clear_msg = users_pipe.clear(params={'user_id': user_id}, debug=debug)
148    if not clear_success:
149        return False, f"Failed to delete user '{user}':\n{clear_msg}"
150    return True, "Success"

Delete a user from the users table.

def get_user_password_hash( self, user: meerschaum.core.User._User.User, debug: bool = False) -> Optional[uuid.UUID]:
153def get_user_password_hash(self, user: User, debug: bool = False) -> Union[uuid.UUID, None]:
154    """
155    Get a user's password hash from the users table.
156    """
157    user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug)
158    users_pipe = self.get_users_pipe()
159    result_df = users_pipe.get_data(['password_hash'], params={'user_id': user_id}, debug=debug)
160    if result_df is None or len(result_df) == 0:
161        return None
162
163    return result_df['password_hash'][0]

Get a user's password hash from the users table.

def get_user_type( self, user: meerschaum.core.User._User.User, debug: bool = False) -> Optional[str]:
166def get_user_type(self, user: User, debug: bool = False) -> Union[str, None]:
167    """
168    Get a user's type from the users table.
169    """
170    user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug)
171    users_pipe = self.get_users_pipe()
172    result_df = users_pipe.get_data(['user_type'], params={'user_id': user_id}, debug=debug)
173    if result_df is None or len(result_df) == 0:
174        return None
175
176    return result_df['user_type'][0]

Get a user's type from the users table.

def get_user_attributes( self, user: meerschaum.core.User._User.User, debug: bool = False) -> Optional[Dict[str, Any]]:
179def get_user_attributes(self, user: User, debug: bool = False) -> Union[Dict[str, Any], None]:
180    """
181    Get a user's attributes from the users table.
182    """
183    user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug)
184    users_pipe = self.get_users_pipe()
185    result_df = users_pipe.get_data(['attributes'], params={'user_id': user_id}, debug=debug)
186    if result_df is None or len(result_df) == 0:
187        return None
188
189    return result_df['attributes'][0]

Get a user's attributes from the users table.

def get_plugins_pipe(self) -> Pipe:
16def get_plugins_pipe(self) -> 'mrsm.Pipe':
17    """
18    Return the internal pipe for syncing plugins metadata.
19    """
20    if '_plugins_pipe' in self.__dict__:
21        return self._plugins_pipe
22
23    cache_connector = self.__dict__.get('_cache_connector', None)
24    users_pipe = self.get_users_pipe()
25    user_id_dtype = users_pipe.dtypes.get('user_id', 'uuid')
26
27    self._plugins_pipe = mrsm.Pipe(
28        'mrsm', 'plugins',
29        instance=self,
30        target='mrsm_plugins',
31        temporary=True,
32        cache=True,
33        cache_connector_keys=cache_connector,
34        static=True,
35        null_indices=False,
36        columns={
37            'primary': 'plugin_name',
38            'user_id': 'user_id',
39        },
40        dtypes={
41            'plugin_name': 'string',
42            'user_id': user_id_dtype,
43            'attributes': 'json',
44            'version': 'string',
45        },
46    )
47    return self._plugins_pipe

Return the internal pipe for syncing plugins metadata.

def register_plugin( self, plugin: Plugin, debug: bool = False) -> Tuple[bool, str]:
50def register_plugin(self, plugin: Plugin, debug: bool = False) -> mrsm.SuccessTuple:
51    """
52    Register a new plugin to the plugins table.
53    """
54    plugins_pipe = self.get_plugins_pipe()
55    users_pipe = self.get_users_pipe()
56    user_id = self.get_plugin_user_id(plugin)
57    if user_id is not None:
58        username = self.get_username(user_id, debug=debug)
59        return False, f"{plugin} is already registered to '{username}'."
60
61    doc = {
62        'plugin_name': plugin.name,
63        'version': plugin.version,
64        'attributes': plugin.attributes,
65        'user_id': plugin.user_id,
66    }
67
68    sync_success, sync_msg = plugins_pipe.sync(
69        [doc],
70        check_existing=False,
71        debug=debug,
72    )
73    if not sync_success:
74        return False, f"Failed to register {plugin}:\n{sync_msg}"
75
76    return True, "Success"

Register a new plugin to the plugins table.

def get_plugin_user_id( self, plugin: Plugin, debug: bool = False) -> Optional[uuid.UUID]:
79def get_plugin_user_id(self, plugin: Plugin, debug: bool = False) -> Union[uuid.UUID, None]:
80    """
81    Return the user ID for plugin's owner.
82    """
83    plugins_pipe = self.get_plugins_pipe() 
84    return plugins_pipe.get_value('user_id', {'plugin_name': plugin.name}, debug=debug)

Return the user ID for plugin's owner.

def delete_plugin( self, plugin: Plugin, debug: bool = False) -> Tuple[bool, str]:
105def delete_plugin(self, plugin: Plugin, debug: bool = False) -> mrsm.SuccessTuple:
106    """
107    Delete a plugin's registration.
108    """
109    plugin_id = self.get_plugin_id(plugin, debug=debug)
110    if plugin_id is None:
111        return False, f"{plugin} is not registered."
112    
113    plugins_pipe = self.get_plugins_pipe()
114    clear_success, clear_msg = plugins_pipe.clear(params={'plugin_name': plugin.name}, debug=debug)
115    if not clear_success:
116        return False, f"Failed to delete {plugin}:\n{clear_msg}"
117    return True, "Success"

Delete a plugin's registration.

def get_plugin_id( self, plugin: Plugin, debug: bool = False) -> Optional[str]:
 97def get_plugin_id(self, plugin: Plugin, debug: bool = False) -> Union[str, None]:
 98    """
 99    Return a plugin's ID.
100    """
101    user_id = self.get_plugin_user_id(plugin, debug=debug)
102    return plugin.name if user_id is not None else None

Return a plugin's ID.

def get_plugin_version( self, plugin: Plugin, debug: bool = False) -> Optional[str]:
120def get_plugin_version(self, plugin: Plugin, debug: bool = False) -> Union[str, None]:
121    """
122    Return the version for a plugin.
123    """
124    plugins_pipe = self.get_plugins_pipe() 
125    return plugins_pipe.get_value('version', {'plugin_name': plugin.name}, debug=debug)

Return the version for a plugin.

def get_plugins( self, user_id: Optional[int] = None, search_term: Optional[str] = None, debug: bool = False, **kw: Any) -> List[str]:
136def get_plugins(
137    self,
138    user_id: Optional[int] = None,
139    search_term: Optional[str] = None,
140    debug: bool = False,
141    **kw: Any
142) -> List[str]:
143    """
144    Return a list of plugin names.
145    """
146    plugins_pipe = self.get_plugins_pipe()
147    params = {}
148    if user_id:
149        params['user_id'] = user_id
150
151    df = plugins_pipe.get_data(['plugin_name'], params=params, debug=debug)
152    if df is None:
153        return []
154
155    docs = df.to_dict(orient='records')
156    return [
157        plugin_name
158        for doc in docs
159        if (plugin_name := doc['plugin_name']).startswith(search_term or '')
160    ]

Return a list of plugin names.

def get_plugin_username( self, plugin: Plugin, debug: bool = False) -> Optional[uuid.UUID]:
87def get_plugin_username(self, plugin: Plugin, debug: bool = False) -> Union[uuid.UUID, None]:
88    """
89    Return the username for plugin's owner.
90    """
91    user_id = self.get_plugin_user_id(plugin, debug=debug)
92    if user_id is None:
93        return None
94    return self.get_username(user_id, debug=debug)

Return the username for plugin's owner.

def get_plugin_attributes( self, plugin: Plugin, debug: bool = False) -> Dict[str, Any]:
128def get_plugin_attributes(self, plugin: Plugin, debug: bool = False) -> Dict[str, Any]:
129    """
130    Return the attributes for a plugin.
131    """
132    plugins_pipe = self.get_plugins_pipe() 
133    return plugins_pipe.get_value('attributes', {'plugin_name': plugin.name}, debug=debug) or {}

Return the attributes for a plugin.

def get_tokens_pipe(self) -> Pipe:
22def get_tokens_pipe(self) -> mrsm.Pipe:
23    """
24    Return the internal pipe for tokens management.
25    """
26    if '_tokens_pipe' in self.__dict__:
27        return self._tokens_pipe
28
29    users_pipe = self.get_users_pipe()
30    user_id_dtype = (
31        users_pipe._attributes.get('parameters', {}).get('dtypes', {}).get('user_id', 'uuid')
32    )
33
34    cache_connector = self.__dict__.get('_cache_connector', None)
35
36    self._tokens_pipe = mrsm.Pipe(
37        'mrsm', 'tokens',
38        instance=self,
39        target='mrsm_tokens',
40        temporary=True,
41        cache=True,
42        cache_connector_keys=cache_connector,
43        static=True,
44        autotime=True,
45        null_indices=False,
46        columns={
47            'datetime': 'creation',
48            'primary': 'id',
49        },
50        indices={
51            'unique': 'label',
52            'user_id': 'user_id',
53        },
54        dtypes={
55            'id': 'uuid',
56            'creation': 'datetime',
57            'expiration': 'datetime',
58            'is_valid': 'bool',
59            'label': 'string',
60            'user_id': user_id_dtype,
61            'scopes': 'json',
62            'secret_hash': 'string',
63        },
64    )
65    return self._tokens_pipe

Return the internal pipe for tokens management.

def register_token( self, token: meerschaum.core.Token._Token.Token, debug: bool = False) -> Tuple[bool, str]:
68def register_token(
69    self,
70    token: Token,
71    debug: bool = False,
72) -> mrsm.SuccessTuple:
73    """
74    Register the new token to the tokens table.
75    """
76    token_id, token_secret = token.generate_credentials()
77    tokens_pipe = self.get_tokens_pipe()
78    user_id = self.get_user_id(token.user) if token.user is not None else None
79    if user_id is None:
80        return False, "Cannot register a token without a user."
81
82    doc = {
83        'id': token_id,
84        'user_id': user_id,
85        'creation': datetime.now(timezone.utc),
86        'expiration': token.expiration,
87        'label': token.label,
88        'is_valid': token.is_valid,
89        'scopes': list(token.scopes) if token.scopes else [],
90        'secret_hash': hash_password(
91            str(token_secret),
92            rounds=STATIC_CONFIG['tokens']['hash_rounds']
93        ),
94    }
95    sync_success, sync_msg = tokens_pipe.sync([doc], check_existing=False, debug=debug)
96    if not sync_success:
97        return False, f"Failed to register token:\n{sync_msg}"
98    return True, "Success"

Register the new token to the tokens table.

def edit_token( self, token: meerschaum.core.Token._Token.Token, debug: bool = False) -> Tuple[bool, str]:
101def edit_token(self, token: Token, debug: bool = False) -> mrsm.SuccessTuple:
102    """
103    Persist the token's in-memory state to the tokens pipe.
104    """
105    if not token.id:
106        return False, "Token ID is not set."
107
108    if not token.exists(debug=debug):
109        return False, f"Token {token.id} does not exist."
110
111    if not token.creation:
112        token_model = self.get_token_model(token.id)
113        token.creation = token_model.creation
114
115    tokens_pipe = self.get_tokens_pipe()
116    doc = {
117        'id': token.id,
118        'creation': token.creation,
119        'expiration': token.expiration,
120        'label': token.label,
121        'is_valid': token.is_valid,
122        'scopes': list(token.scopes) if token.scopes else [],
123    }
124    sync_success, sync_msg = tokens_pipe.sync([doc], debug=debug)
125    if not sync_success:
126        return False, f"Failed to edit token '{token.id}':\n{sync_msg}"
127
128    return True, "Success"

Persist the token's in-memory state to the tokens pipe.

def invalidate_token( self, token: meerschaum.core.Token._Token.Token, debug: bool = False) -> Tuple[bool, str]:
131def invalidate_token(self, token: Token, debug: bool = False) -> mrsm.SuccessTuple:
132    """
133    Set `is_valid` to `False` for the given token.
134    """
135    if not token.id:
136        return False, "Token ID is not set."
137
138    if not token.exists(debug=debug):
139        return False, f"Token {token.id} does not exist."
140
141    if not token.creation:
142        token_model = self.get_token_model(token.id)
143        token.creation = token_model.creation
144
145    token.is_valid = False
146    tokens_pipe = self.get_tokens_pipe()
147    doc = {
148        'id': token.id,
149        'creation': token.creation,
150        'is_valid': False,
151    }
152    sync_success, sync_msg = tokens_pipe.sync([doc], debug=debug)
153    if not sync_success:
154        return False, f"Failed to invalidate token '{token.id}':\n{sync_msg}"
155
156    return True, "Success"

Set is_valid to False for the given token.

def delete_token( self, token: meerschaum.core.Token._Token.Token, debug: bool = False) -> Tuple[bool, str]:
159def delete_token(self, token: Token, debug: bool = False) -> mrsm.SuccessTuple:
160    """
161    Delete the given token from the tokens table.
162    """
163    if not token.id:
164        return False, "Token ID is not set."
165
166    if not token.exists(debug=debug):
167        return False, f"Token {token.id} does not exist."
168
169    if not token.creation:
170        token_model = self.get_token_model(token.id)
171        token.creation = token_model.creation
172
173    token.is_valid = False
174    tokens_pipe = self.get_tokens_pipe()
175    clear_success, clear_msg = tokens_pipe.clear(params={'id': token.id}, debug=debug)
176    if not clear_success:
177        return False, f"Failed to delete token '{token.id}':\n{clear_msg}"
178
179    return True, "Success"

Delete the given token from the tokens table.

def get_token( self, token_id: Union[uuid.UUID, str], debug: bool = False) -> Optional[meerschaum.core.Token._Token.Token]:
235def get_token(self, token_id: Union[uuid.UUID, str], debug: bool = False) -> Union[Token, None]:
236    """
237    Return the `Token` from its ID.
238    """
239    from meerschaum.utils.misc import is_uuid
240    if isinstance(token_id, str):
241        if is_uuid(token_id):
242            token_id = uuid.UUID(token_id)
243        else:
244            raise ValueError("Invalid token ID.")
245    token_model = self.get_token_model(token_id)
246    if token_model is None:
247        return None
248    return Token(**dict(token_model))

Return the Token from its ID.

def get_tokens( self, user: Optional[meerschaum.core.User._User.User] = None, labels: Optional[List[str]] = None, ids: Optional[List[uuid.UUID]] = None, debug: bool = False) -> List[meerschaum.core.Token._Token.Token]:
182def get_tokens(
183    self,
184    user: Optional[User] = None,
185    labels: Optional[List[str]] = None,
186    ids: Optional[List[uuid.UUID]] = None,
187    debug: bool = False,
188) -> List[Token]:
189    """
190    Return a list of `Token` objects.
191    """
192    tokens_pipe = self.get_tokens_pipe()
193    user_id = (
194        self.get_user_id(user, debug=debug)
195        if user is not None
196        else None
197    )
198    user_type = self.get_user_type(user, debug=debug) if user is not None else None
199    params = (
200        {
201            'user_id': (
202                user_id
203                if user_type != 'admin'
204                else [user_id, None]
205            )
206        }
207        if user_id is not None
208        else {}
209    )
210    if labels:
211        params['label'] = labels
212    if ids:
213        params['id'] = ids
214        
215    if debug:
216        dprint(f"Getting tokens with {user_id=}, {params=}")
217
218    tokens_df = tokens_pipe.get_data(params=params, debug=debug)
219    if tokens_df is None:
220        return []
221
222    if debug:
223        dprint(f"Retrieved tokens dataframe:\n{tokens_df}")
224
225    tokens_docs = tokens_df.to_dict(orient='records')
226    return [
227        Token(
228            instance=self,
229            **token_doc
230        )
231        for token_doc in reversed(tokens_docs)
232    ]

Return a list of Token objects.

def get_token_model( self, token_id: Union[uuid.UUID, meerschaum.core.Token._Token.Token], debug: bool = False) -> "'Union[TokenModel, None]'":
251def get_token_model(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> 'Union[TokenModel, None]':
252    """
253    Return a token's model from the instance.
254    """
255    from meerschaum.models import TokenModel
256    if isinstance(token_id, Token):
257        token_id = Token.id
258    if not token_id:
259        raise ValueError("Invalid token ID.")
260    tokens_pipe = self.get_tokens_pipe()
261    doc = tokens_pipe.get_doc(
262        params={'id': token_id},
263        debug=debug,
264    )
265    if doc is None:
266        return None
267    return TokenModel(**doc)

Return a token's model from the instance.

def get_token_secret_hash( self, token_id: Union[uuid.UUID, meerschaum.core.Token._Token.Token], debug: bool = False) -> Optional[str]:
270def get_token_secret_hash(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> Union[str, None]:
271    """
272    Return the secret hash for a given token.
273    """
274    if isinstance(token_id, Token):
275        token_id = token_id.id
276    if not token_id:
277        raise ValueError("Invalid token ID.")
278    tokens_pipe = self.get_tokens_pipe()
279    return tokens_pipe.get_value('secret_hash', params={'id': token_id}, debug=debug)

Return the secret hash for a given token.

def token_exists( self, token_id: Union[uuid.UUID, meerschaum.core.Token._Token.Token], debug: bool = False) -> bool:
308def token_exists(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> bool:
309    """
310    Return `True` if a token exists in the tokens pipe.
311    """
312    if isinstance(token_id, Token):
313        token_id = token_id.id
314    if not token_id:
315        raise ValueError("Invalid token ID.")
316
317    tokens_pipe = self.get_tokens_pipe()
318    return tokens_pipe.get_value('creation', params={'id': token_id}, debug=debug) is not None

Return True if a token exists in the tokens pipe.

def get_token_scopes( self, token_id: Union[uuid.UUID, meerschaum.core.Token._Token.Token], debug: bool = False) -> List[str]:
295def get_token_scopes(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> List[str]:
296    """
297    Return the scopes for a token.
298    """
299    if isinstance(token_id, Token):
300        token_id = token_id.id
301    if not token_id:
302        raise ValueError("Invalid token ID.")
303
304    tokens_pipe = self.get_tokens_pipe()
305    return tokens_pipe.get_value('scopes', params={'id': token_id}, debug=debug) or []

Return the scopes for a token.

@abc.abstractmethod
def register_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
17@abc.abstractmethod
18def register_pipe(
19    self,
20    pipe: mrsm.Pipe,
21    debug: bool = False,
22    **kwargs: Any
23) -> mrsm.SuccessTuple:
24    """
25    Insert the pipe's attributes into the internal `pipes` table.
26
27    Parameters
28    ----------
29    pipe: mrsm.Pipe
30        The pipe to be registered.
31
32    Returns
33    -------
34    A `SuccessTuple` of the result.
35    """

Insert the pipe's attributes into the internal pipes table.

Parameters
  • pipe (mrsm.Pipe): The pipe to be registered.
Returns
@abc.abstractmethod
def get_pipe_attributes( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Dict[str, Any]:
37@abc.abstractmethod
38def get_pipe_attributes(
39    self,
40    pipe: mrsm.Pipe,
41    debug: bool = False,
42    **kwargs: Any
43) -> Dict[str, Any]:
44    """
45    Return the pipe's document from the internal `pipes` table.
46
47    Parameters
48    ----------
49    pipe: mrsm.Pipe
50        The pipe whose attributes should be retrieved.
51
52    Returns
53    -------
54    The document that matches the keys of the pipe.
55    """

Return the pipe's document from the internal pipes table.

Parameters
  • pipe (mrsm.Pipe): The pipe whose attributes should be retrieved.
Returns
  • The document that matches the keys of the pipe.
@abc.abstractmethod
def get_pipe_id( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Union[str, int, NoneType]:
57@abc.abstractmethod
58def get_pipe_id(
59    self,
60    pipe: mrsm.Pipe,
61    debug: bool = False,
62    **kwargs: Any
63) -> Union[str, int, None]:
64    """
65    Return the `id` for the pipe if it exists.
66
67    Parameters
68    ----------
69    pipe: mrsm.Pipe
70        The pipe whose `id` to fetch.
71
72    Returns
73    -------
74    The `id` for the pipe's document or `None`.
75    """

Return the id for the pipe if it exists.

Parameters
  • pipe (mrsm.Pipe): The pipe whose id to fetch.
Returns
  • The id for the pipe's document or None.
def edit_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
77def edit_pipe(
78    self,
79    pipe: mrsm.Pipe,
80    debug: bool = False,
81    **kwargs: Any
82) -> mrsm.SuccessTuple:
83    """
84    Edit the attributes of the pipe.
85
86    Parameters
87    ----------
88    pipe: mrsm.Pipe
89        The pipe whose in-memory parameters must be persisted.
90
91    Returns
92    -------
93    A `SuccessTuple` indicating success.
94    """
95    raise NotImplementedError

Edit the attributes of the pipe.

Parameters
  • pipe (mrsm.Pipe): The pipe whose in-memory parameters must be persisted.
Returns
def delete_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
 97def delete_pipe(
 98    self,
 99    pipe: mrsm.Pipe,
100    debug: bool = False,
101    **kwargs: Any
102) -> mrsm.SuccessTuple:
103    """
104    Delete a pipe's registration from the `pipes` collection.
105
106    Parameters
107    ----------
108    pipe: mrsm.Pipe
109        The pipe to be deleted.
110
111    Returns
112    -------
113    A `SuccessTuple` indicating success.
114    """
115    raise NotImplementedError

Delete a pipe's registration from the pipes collection.

Parameters
  • pipe (mrsm.Pipe): The pipe to be deleted.
Returns
@abc.abstractmethod
def fetch_pipes_keys( self, connector_keys: Optional[List[str]] = None, metric_keys: Optional[List[str]] = None, location_keys: Optional[List[str]] = None, tags: Optional[List[str]] = None, debug: bool = False, **kwargs: Any) -> Union[List[Tuple[str, str, str]], List[Tuple[str, str, str, Union[Dict[str, Any], List[str]]]], Dict[Union[int, str], Tuple[str, str, str]], Dict[Union[int, str], Tuple[str, str, str, Union[Dict[str, Any], List[str]]]]]:
117@abc.abstractmethod
118def fetch_pipes_keys(
119    self,
120    connector_keys: Optional[List[str]] = None,
121    metric_keys: Optional[List[str]] = None,
122    location_keys: Optional[List[str]] = None,
123    tags: Optional[List[str]] = None,
124    debug: bool = False,
125    **kwargs: Any
126) -> Union[
127    List[Tuple[str, str, str]],
128    List[Tuple[str, str, str, Union[Dict[str, Any], List[str]]]],
129    Dict[Union[int, str], Tuple[str, str, str]],
130    Dict[Union[int, str], Tuple[str, str, str, Union[Dict[str, Any], List[str]]]],
131]:
132    """
133    Return registered pipes' keys according to the provided filters.
134
135    May return either a list of key tuples or a dictionary mapping pipe IDs to key tuples.
136    When returning a dictionary, the key is the pipe's unique ID (int or str).
137    Tuples may be length 3 `(connector_keys, metric_key, location_key)` or length 4
138    with parameters or tags appended as the fourth element.
139
140    Parameters
141    ----------
142    connector_keys: list[str] | None, default None
143        The keys passed via `-c`.
144
145    metric_keys: list[str] | None, default None
146        The keys passed via `-m`.
147
148    location_keys: list[str] | None, default None
149        The keys passed via `-l`.
150
151    tags: List[str] | None, default None
152        Tags passed via `--tags` which are stored under `parameters:tags`.
153
154    Returns
155    -------
156    A list of tuples or a dictionary mapping pipe IDs to tuples.
157    You may return the string `"None"` for location keys in place of nulls.
158
159    Examples
160    --------
161    >>> import meerschaum as mrsm
162    >>> conn = mrsm.get_connector('example:demo')
163    >>>
164    >>> pipe_a = mrsm.Pipe('a', 'demo', tags=['foo'], instance=conn)
165    >>> pipe_b = mrsm.Pipe('b', 'demo', tags=['bar'], instance=conn)
166    >>> pipe_a.register()
167    >>> pipe_b.register()
168    >>>
169    >>> conn.fetch_pipes_keys(['a', 'b'])
170    [('a', 'demo', 'None'), ('b', 'demo', 'None')]
171    >>> conn.fetch_pipes_keys(metric_keys=['demo'])
172    [('a', 'demo', 'None'), ('b', 'demo', 'None')]
173    >>> conn.fetch_pipes_keys(tags=['foo'])
174    [('a', 'demo', 'None')]
175    >>> conn.fetch_pipes_keys(location_keys=[None])
176    [('a', 'demo', 'None'), ('b', 'demo', 'None')]
177    """

Return registered pipes' keys according to the provided filters.

May return either a list of key tuples or a dictionary mapping pipe IDs to key tuples. When returning a dictionary, the key is the pipe's unique ID (int or str). Tuples may be length 3 (connector_keys, metric_key, location_key) or length 4 with parameters or tags appended as the fourth element.

Parameters
  • connector_keys (list[str] | None, default None): The keys passed via -c.
  • metric_keys (list[str] | None, default None): The keys passed via -m.
  • location_keys (list[str] | None, default None): The keys passed via -l.
  • tags (List[str] | None, default None): Tags passed via --tags which are stored under parameters:tags.
Returns
  • A list of tuples or a dictionary mapping pipe IDs to tuples.
  • You may return the string "None" for location keys in place of nulls.
Examples
>>> import meerschaum as mrsm
>>> conn = mrsm.get_connector('example:demo')
>>>
>>> pipe_a = mrsm.Pipe('a', 'demo', tags=['foo'], instance=conn)
>>> pipe_b = mrsm.Pipe('b', 'demo', tags=['bar'], instance=conn)
>>> pipe_a.register()
>>> pipe_b.register()
>>>
>>> conn.fetch_pipes_keys(['a', 'b'])
[('a', 'demo', 'None'), ('b', 'demo', 'None')]
>>> conn.fetch_pipes_keys(metric_keys=['demo'])
[('a', 'demo', 'None'), ('b', 'demo', 'None')]
>>> conn.fetch_pipes_keys(tags=['foo'])
[('a', 'demo', 'None')]
>>> conn.fetch_pipes_keys(location_keys=[None])
[('a', 'demo', 'None'), ('b', 'demo', 'None')]
@abc.abstractmethod
def pipe_exists( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> bool:
179@abc.abstractmethod
180def pipe_exists(
181    self,
182    pipe: mrsm.Pipe,
183    debug: bool = False,
184    **kwargs: Any
185) -> bool:
186    """
187    Check whether a pipe's target table exists.
188
189    Parameters
190    ----------
191    pipe: mrsm.Pipe
192        The pipe to check whether its table exists.
193
194    Returns
195    -------
196    A `bool` indicating the table exists.
197    """

Check whether a pipe's target table exists.

Parameters
  • pipe (mrsm.Pipe): The pipe to check whether its table exists.
Returns
  • A bool indicating the table exists.
@abc.abstractmethod
def drop_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
199@abc.abstractmethod
200def drop_pipe(
201    self,
202    pipe: mrsm.Pipe,
203    debug: bool = False,
204    **kwargs: Any
205) -> mrsm.SuccessTuple:
206    """
207    Drop a pipe's collection if it exists.
208
209    Parameters
210    ----------
211    pipe: mrsm.Pipe
212        The pipe to be dropped.
213
214    Returns
215    -------
216    A `SuccessTuple` indicating success.
217    """
218    raise NotImplementedError

Drop a pipe's collection if it exists.

Parameters
  • pipe (mrsm.Pipe): The pipe to be dropped.
Returns
def drop_pipe_indices( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
220def drop_pipe_indices(
221    self,
222    pipe: mrsm.Pipe,
223    debug: bool = False,
224    **kwargs: Any
225) -> mrsm.SuccessTuple:
226    """
227    Drop a pipe's indices.
228
229    Parameters
230    ----------
231    pipe: mrsm.Pipe
232        The pipe whose indices need to be dropped.
233
234    Returns
235    -------
236    A `SuccessTuple` indicating success.
237    """
238    return False, f"Cannot drop indices for instance connectors of type '{self.type}'."

Drop a pipe's indices.

Parameters
  • pipe (mrsm.Pipe): The pipe whose indices need to be dropped.
Returns
@abc.abstractmethod
def sync_pipe( self, pipe: Pipe, df: "'pd.DataFrame'" = None, begin: Union[datetime.datetime, int, NoneType] = None, end: Union[datetime.datetime, int, NoneType] = None, chunksize: Optional[int] = -1, check_existing: bool = True, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
240@abc.abstractmethod
241def sync_pipe(
242    self,
243    pipe: mrsm.Pipe,
244    df: 'pd.DataFrame' = None,
245    begin: Union[datetime, int, None] = None,
246    end: Union[datetime, int, None] = None,
247    chunksize: Optional[int] = -1,
248    check_existing: bool = True,
249    debug: bool = False,
250    **kwargs: Any
251) -> mrsm.SuccessTuple:
252    """
253    Sync a pipe using a database connection.
254
255    Parameters
256    ----------
257    pipe: mrsm.Pipe
258        The Meerschaum Pipe instance into which to sync the data.
259
260    df: Optional[pd.DataFrame]
261        An optional DataFrame or equivalent to sync into the pipe.
262        Defaults to `None`.
263
264    begin: Union[datetime, int, None], default None
265        Optionally specify the earliest datetime to search for data.
266        Defaults to `None`.
267
268    end: Union[datetime, int, None], default None
269        Optionally specify the latest datetime to search for data.
270        Defaults to `None`.
271
272    chunksize: Optional[int], default -1
273        Specify the number of rows to sync per chunk.
274        If `-1`, resort to system configuration (default is `900`).
275        A `chunksize` of `None` will sync all rows in one transaction.
276        Defaults to `-1`.
277
278    check_existing: bool, default True
279        If `True`, pull and diff with existing data from the pipe. Defaults to `True`.
280
281    debug: bool, default False
282        Verbosity toggle. Defaults to False.
283
284    Returns
285    -------
286    A `SuccessTuple` of success (`bool`) and message (`str`).
287    """

Sync a pipe using a database connection.

Parameters
  • pipe (mrsm.Pipe): The Meerschaum Pipe instance into which to sync the data.
  • df (Optional[pd.DataFrame]): An optional DataFrame or equivalent to sync into the pipe. Defaults to None.
  • begin (Union[datetime, int, None], default None): Optionally specify the earliest datetime to search for data. Defaults to None.
  • end (Union[datetime, int, None], default None): Optionally specify the latest datetime to search for data. Defaults to None.
  • chunksize (Optional[int], default -1): Specify the number of rows to sync per chunk. If -1, resort to system configuration (default is 900). A chunksize of None will sync all rows in one transaction. Defaults to -1.
  • check_existing (bool, default True): If True, pull and diff with existing data from the pipe. Defaults to True.
  • debug (bool, default False): Verbosity toggle. Defaults to False.
Returns
def create_pipe_indices( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
289def create_pipe_indices(
290    self,
291    pipe: mrsm.Pipe,
292    debug: bool = False,
293    **kwargs: Any
294) -> mrsm.SuccessTuple:
295    """
296    Create a pipe's indices.
297
298    Parameters
299    ----------
300    pipe: mrsm.Pipe
301        The pipe whose indices need to be created.
302
303    Returns
304    -------
305    A `SuccessTuple` indicating success.
306    """
307    return False, f"Cannot create indices for instance connectors of type '{self.type}'."

Create a pipe's indices.

Parameters
  • pipe (mrsm.Pipe): The pipe whose indices need to be created.
Returns
def clear_pipe( self, pipe: Pipe, begin: Union[datetime.datetime, int, NoneType] = None, end: Union[datetime.datetime, int, NoneType] = None, params: Optional[Dict[str, Any]] = None, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
309def clear_pipe(
310    self,
311    pipe: mrsm.Pipe,
312    begin: Union[datetime, int, None] = None,
313    end: Union[datetime, int, None] = None,
314    params: Optional[Dict[str, Any]] = None,
315    debug: bool = False,
316    **kwargs: Any
317) -> mrsm.SuccessTuple:
318    """
319    Delete rows within `begin`, `end`, and `params`.
320
321    Parameters
322    ----------
323    pipe: mrsm.Pipe
324        The pipe whose rows to clear.
325
326    begin: datetime | int | None, default None
327        If provided, remove rows >= `begin`.
328
329    end: datetime | int | None, default None
330        If provided, remove rows < `end`.
331
332    params: dict[str, Any] | None, default None
333        If provided, only remove rows which match the `params` filter.
334
335    Returns
336    -------
337    A `SuccessTuple` indicating success.
338    """
339    raise NotImplementedError

Delete rows within begin, end, and params.

Parameters
  • pipe (mrsm.Pipe): The pipe whose rows to clear.
  • begin (datetime | int | None, default None): If provided, remove rows >= begin.
  • end (datetime | int | None, default None): If provided, remove rows < end.
  • params (dict[str, Any] | None, default None): If provided, only remove rows which match the params filter.
Returns
def get_pipe_data( self, pipe: Pipe, select_columns: Optional[List[str]] = None, omit_columns: Optional[List[str]] = None, begin: Union[datetime.datetime, int, NoneType] = None, end: Union[datetime.datetime, int, NoneType] = None, params: Optional[Dict[str, Any]] = None, debug: bool = False, **kwargs: Any) -> "Union['pd.DataFrame', None]":
341def get_pipe_data(
342    self,
343    pipe: mrsm.Pipe,
344    select_columns: Optional[List[str]] = None,
345    omit_columns: Optional[List[str]] = None,
346    begin: Union[datetime, int, None] = None,
347    end: Union[datetime, int, None] = None,
348    params: Optional[Dict[str, Any]] = None,
349    debug: bool = False,
350    **kwargs: Any
351) -> Union['pd.DataFrame', None]:
352    """
353    Query a pipe's target table and return the DataFrame.
354
355    Parameters
356    ----------
357    pipe: mrsm.Pipe
358        The pipe with the target table from which to read.
359
360    select_columns: list[str] | None, default None
361        If provided, only select these given columns.
362        Otherwise select all available columns (i.e. `SELECT *`).
363
364    omit_columns: list[str] | None, default None
365        If provided, remove these columns from the selection.
366
367    begin: datetime | int | None, default None
368        The earliest `datetime` value to search from (inclusive).
369
370    end: datetime | int | None, default None
371        The lastest `datetime` value to search from (exclusive).
372
373    params: dict[str | str] | None, default None
374        Additional filters to apply to the query.
375
376    Returns
377    -------
378    The target table's data as a DataFrame.
379    """
380    if type(self).get_pipe_docs is get_pipe_docs:
381        raise NotImplementedError(
382            f"Missing `get_pipe_data()` or `get_pipe_docs()` for {type(self)}."
383        )
384
385    docs = self.get_pipe_docs(
386        pipe=pipe,
387        select_columns=select_columns,
388        omit_columns=omit_columns,
389        begin=begin,
390        end=end,
391        params=params,
392        debug=debug,
393        **kwargs
394    )
395    if not docs:
396        return None
397
398    pd = mrsm.attempt_import('pandas')
399    try:
400        return pd.DataFrame(docs)
401    except Exception as e:
402        from meerschaum.utils.warnings import warn
403        warn(f"Cannot build DataFrame from pipe docs:\n{e}")
404    
405    return None

Query a pipe's target table and return the DataFrame.

Parameters
  • pipe (mrsm.Pipe): The pipe with the target table from which to read.
  • select_columns (list[str] | None, default None): If provided, only select these given columns. Otherwise select all available columns (i.e. SELECT *).
  • omit_columns (list[str] | None, default None): If provided, remove these columns from the selection.
  • begin (datetime | int | None, default None): The earliest datetime value to search from (inclusive).
  • end (datetime | int | None, default None): The lastest datetime value to search from (exclusive).
  • params (dict[str | str] | None, default None): Additional filters to apply to the query.
Returns
  • The target table's data as a DataFrame.
def get_pipe_docs( self, pipe: Pipe, select_columns: Optional[List[str]] = None, omit_columns: Optional[List[str]] = None, begin: Union[datetime.datetime, int, NoneType] = None, end: Union[datetime.datetime, int, NoneType] = None, params: Optional[Dict[str, Any]] = None, debug: bool = False, **kwargs: Any) -> list[dict[str, typing.Any]]:
407def get_pipe_docs(
408    self,
409    pipe: mrsm.Pipe,
410    select_columns: Optional[List[str]] = None,
411    omit_columns: Optional[List[str]] = None,
412    begin: Union[datetime, int, None] = None,
413    end: Union[datetime, int, None] = None,
414    params: Optional[Dict[str, Any]] = None,
415    debug: bool = False,
416    **kwargs: Any
417) -> list[dict[str, Any]]:
418    """
419    Return a pipe's data as a list of documents.
420    Defaults to `get_pipe_data().to_dict(orient='records')`.
421
422    Parameters
423    ----------
424    pipe: mrsm.Pipe
425        The pipe with the target table from which to read.
426
427    select_columns: list[str] | None, default None
428        If provided, only select these given columns.
429        Otherwise select all available columns (i.e. `SELECT *`).
430
431    omit_columns: list[str] | None, default None
432        If provided, remove these columns from the selection.
433
434    begin: datetime | int | None, default None
435        The earliest `datetime` value to search from (inclusive).
436
437    end: datetime | int | None, default None
438        The lastest `datetime` value to search from (exclusive).
439
440    params: dict[str | str] | None, default None
441        Additional filters to apply to the query.
442
443    Returns
444    -------
445    The target table's data as a list of dictionaries.
446    """
447    df = self.get_pipe_data(
448        pipe=pipe,
449        select_columns=select_columns,
450        omit_columns=omit_columns,
451        begin=begin,
452        end=end,
453        params=params,
454        debug=debug,
455        **kwargs
456    )
457    if df is None or df.empty:
458        return []
459    return df.to_dict(orient='records')

Return a pipe's data as a list of documents. Defaults to get_pipe_data().to_dict(orient='records').

Parameters
  • pipe (mrsm.Pipe): The pipe with the target table from which to read.
  • select_columns (list[str] | None, default None): If provided, only select these given columns. Otherwise select all available columns (i.e. SELECT *).
  • omit_columns (list[str] | None, default None): If provided, remove these columns from the selection.
  • begin (datetime | int | None, default None): The earliest datetime value to search from (inclusive).
  • end (datetime | int | None, default None): The lastest datetime value to search from (exclusive).
  • params (dict[str | str] | None, default None): Additional filters to apply to the query.
Returns
  • The target table's data as a list of dictionaries.
@abc.abstractmethod
def get_sync_time( self, pipe: Pipe, params: Optional[Dict[str, Any]] = None, newest: bool = True, debug: bool = False, **kwargs: Any) -> datetime.datetime | int | None:
461@abc.abstractmethod
462def get_sync_time(
463    self,
464    pipe: mrsm.Pipe,
465    params: Optional[Dict[str, Any]] = None,
466    newest: bool = True,
467    debug: bool = False,
468    **kwargs: Any
469) -> datetime | int | None:
470    """
471    Return the most recent value for the `datetime` axis.
472
473    Parameters
474    ----------
475    pipe: mrsm.Pipe
476        The pipe whose collection contains documents.
477
478    params: dict[str, Any] | None, default None
479        Filter certain parameters when determining the sync time.
480
481    newest: bool, default True
482        If `True`, return the maximum value for the column.
483
484    Returns
485    -------
486    The largest `datetime` or `int` value of the `datetime` axis. 
487    """

Return the most recent value for the datetime axis.

Parameters
  • pipe (mrsm.Pipe): The pipe whose collection contains documents.
  • params (dict[str, Any] | None, default None): Filter certain parameters when determining the sync time.
  • newest (bool, default True): If True, return the maximum value for the column.
Returns
  • The largest datetime or int value of the datetime axis.
@abc.abstractmethod
def get_pipe_columns_types( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Dict[str, str]:
489@abc.abstractmethod
490def get_pipe_columns_types(
491    self,
492    pipe: mrsm.Pipe,
493    debug: bool = False,
494    **kwargs: Any
495) -> Dict[str, str]:
496    """
497    Return the data types for the columns in the target table for data type enforcement.
498
499    Parameters
500    ----------
501    pipe: mrsm.Pipe
502        The pipe whose target table contains columns and data types.
503
504    Returns
505    -------
506    A dictionary mapping columns to data types.
507    """

Return the data types for the columns in the target table for data type enforcement.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table contains columns and data types.
Returns
  • A dictionary mapping columns to data types.
def get_pipe_columns_indices(self, debug: bool = False) -> Dict[str, List[Dict[str, str]]]:
509def get_pipe_columns_indices(
510    self,
511    debug: bool = False,
512) -> Dict[str, List[Dict[str, str]]]:
513    """
514    Return a dictionary mapping columns to metadata about related indices.
515
516    Parameters
517    ----------
518    pipe: mrsm.Pipe
519        The pipe whose target table has related indices.
520
521    Returns
522    -------
523    A list of dictionaries with the keys "type" and "name".
524
525    Examples
526    --------
527    >>> pipe = mrsm.Pipe('demo', 'shirts', columns={'primary': 'id'}, indices={'size_color': ['color', 'size']})
528    >>> pipe.sync([{'color': 'red', 'size': 'M'}])
529    >>> pipe.get_columns_indices()
530    {'id': [{'name': 'demo_shirts_pkey', 'type': 'PRIMARY KEY'}], 'color': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}], 'size': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}]}
531    """
532    return {}

Return a dictionary mapping columns to metadata about related indices.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table has related indices.
Returns
  • A list of dictionaries with the keys "type" and "name".
Examples
>>> pipe = mrsm.Pipe('demo', 'shirts', columns={'primary': 'id'}, indices={'size_color': ['color', 'size']})
>>> pipe.sync([{'color': 'red', 'size': 'M'}])
>>> pipe.get_columns_indices()
{'id': [{'name': 'demo_shirts_pkey', 'type': 'PRIMARY KEY'}], 'color': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}], 'size': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}]}
def get_pipe_size( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Optional[int]:
534def get_pipe_size(
535    self,
536    pipe: mrsm.Pipe,
537    debug: bool = False,
538    **kwargs: Any
539) -> Union[int, None]:
540    """
541    Return the on-disk size of a pipe's target table in bytes.
542
543    Parameters
544    ----------
545    pipe: mrsm.Pipe
546        The pipe whose target table size to measure.
547
548    Returns
549    -------
550    An `int` of the number of bytes occupied by the target table,
551    or `None` if the size cannot be determined.
552    """
553    raise NotImplementedError(
554        f"`get_pipe_size()` is not implemented for instance connectors of type '{self.type}'."
555    )

Return the on-disk size of a pipe's target table in bytes.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table size to measure.
Returns
  • An int of the number of bytes occupied by the target table,
  • or None if the size cannot be determined.
def compress_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
557def compress_pipe(
558    self,
559    pipe: mrsm.Pipe,
560    debug: bool = False,
561    **kwargs: Any
562) -> mrsm.SuccessTuple:
563    """
564    Compress a pipe's target table to reduce disk usage.
565
566    Parameters
567    ----------
568    pipe: mrsm.Pipe
569        The pipe whose target table to compress.
570
571    Returns
572    -------
573    A `SuccessTuple` indicating success.
574    """
575    return False, (
576        f"Compression is not supported for instance connectors of type '{self.type}'."
577    )

Compress a pipe's target table to reduce disk usage.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table to compress.
Returns
def decompress_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
579def decompress_pipe(
580    self,
581    pipe: mrsm.Pipe,
582    debug: bool = False,
583    **kwargs: Any
584) -> mrsm.SuccessTuple:
585    """
586    Decompress a pipe's target table, the inverse of `compress_pipe()`.
587
588    Parameters
589    ----------
590    pipe: mrsm.Pipe
591        The pipe whose target table to decompress.
592
593    Returns
594    -------
595    A `SuccessTuple` indicating success.
596    """
597    return False, (
598        f"Decompression is not supported for instance connectors of type '{self.type}'."
599    )

Decompress a pipe's target table, the inverse of compress_pipe().

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table to decompress.
Returns
def vacuum_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
601def vacuum_pipe(
602    self,
603    pipe: mrsm.Pipe,
604    debug: bool = False,
605    **kwargs: Any
606) -> mrsm.SuccessTuple:
607    """
608    Reclaim disk space from a pipe's target table.
609
610    Parameters
611    ----------
612    pipe: mrsm.Pipe
613        The pipe whose target table to vacuum.
614
615    Returns
616    -------
617    A `SuccessTuple` indicating success.
618    """
619    return False, (
620        f"Vacuuming is not supported for instance connectors of type '{self.type}'."
621    )

Reclaim disk space from a pipe's target table.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table to vacuum.
Returns
def analyze_pipe( self, pipe: Pipe, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
623def analyze_pipe(
624    self,
625    pipe: mrsm.Pipe,
626    debug: bool = False,
627    **kwargs: Any
628) -> mrsm.SuccessTuple:
629    """
630    Refresh the planner statistics for a pipe's target table.
631
632    Parameters
633    ----------
634    pipe: mrsm.Pipe
635        The pipe whose target table to analyze.
636
637    Returns
638    -------
639    A `SuccessTuple` indicating success.
640    """
641    return False, (
642        f"Analyzing is not supported for instance connectors of type '{self.type}'."
643    )

Refresh the planner statistics for a pipe's target table.

Parameters
  • pipe (mrsm.Pipe): The pipe whose target table to analyze.
Returns
def partition_pipe( self, pipe: Pipe, chunk_minutes: Optional[int] = None, debug: bool = False, **kwargs: Any) -> Tuple[bool, str]:
645def partition_pipe(
646    self,
647    pipe: mrsm.Pipe,
648    chunk_minutes: Optional[int] = None,
649    debug: bool = False,
650    **kwargs: Any
651) -> mrsm.SuccessTuple:
652    """
653    Rebuild a pipe's target table to a new partition (chunk) width.
654
655    Parameters
656    ----------
657    pipe: mrsm.Pipe
658        The partitioned pipe whose target table to repartition.
659
660    chunk_minutes: Optional[int], default None
661        The new partition width in minutes. Defaults to the pipe's `verify.chunk_minutes`.
662
663    Returns
664    -------
665    A `SuccessTuple` indicating success.
666    """
667    return False, (
668        f"Repartitioning is not supported for instance connectors of type '{self.type}'."
669    )

Rebuild a pipe's target table to a new partition (chunk) width.

Parameters
  • pipe (mrsm.Pipe): The partitioned pipe whose target table to repartition.
  • chunk_minutes (Optional[int], default None): The new partition width in minutes. Defaults to the pipe's verify.chunk_minutes.
Returns
def make_connector(cls, _is_executor: bool = False):
279def make_connector(cls, _is_executor: bool = False):
280    """
281    Register a class as a `Connector`.
282    The `type` will be the lower case of the class name, without the suffix `connector`.
283
284    Parameters
285    ----------
286    instance: bool, default False
287        If `True`, make this connector type an instance connector.
288        This requires implementing the various pipes functions and lots of testing.
289
290    Examples
291    --------
292    >>> import meerschaum as mrsm
293    >>> from meerschaum.connectors import make_connector, Connector
294    >>> 
295    >>> @make_connector
296    >>> class FooConnector(Connector):
297    ...     REQUIRED_ATTRIBUTES: list[str] = ['username', 'password']
298    ... 
299    >>> conn = mrsm.get_connector('foo:bar', username='dog', password='cat')
300    >>> print(conn.username, conn.password)
301    dog cat
302    >>> 
303    """
304    import re
305    from meerschaum.plugins import _get_parent_plugin
306    suffix_regex = (
307        r'connector$'
308        if not _is_executor
309        else r'executor$'
310    )
311    plugin_name = _get_parent_plugin(2)
312    typ = re.sub(suffix_regex, '', cls.__name__.lower())
313    with _locks['types']:
314        types[typ] = cls
315    with _locks['custom_types']:
316        custom_types.add(typ)
317    if plugin_name:
318        with _locks['plugins_types']:
319            if plugin_name not in plugins_types:
320                plugins_types[plugin_name] = []
321            plugins_types[plugin_name].append(typ)
322    with _locks['connectors']:
323        if typ not in connectors:
324            connectors[typ] = {}
325    if getattr(cls, 'IS_INSTANCE', False):
326        with _locks['instance_types']:
327            if typ not in instance_types:
328                instance_types.append(typ)
329
330    return cls

Register a class as a Connector. The type will be the lower case of the class name, without the suffix connector.

Parameters
  • instance (bool, default False): If True, make this connector type an instance connector. This requires implementing the various pipes functions and lots of testing.
Examples
>>> import meerschaum as mrsm
>>> from meerschaum.connectors import make_connector, Connector
>>> 
>>> @make_connector
>>> class FooConnector(Connector):
...     REQUIRED_ATTRIBUTES: list[str] = ['username', 'password']
... 
>>> conn = mrsm.get_connector('foo:bar', username='dog', password='cat')
>>> print(conn.username, conn.password)
dog cat
>>>
def entry( sysargs: Union[List[str], str, NoneType] = None, _patch_args: Optional[Dict[str, Any]] = None, _use_cli_daemon: bool = True, _session_id: Optional[str] = None) -> Tuple[bool, str]:
53def entry(
54    sysargs: Union[List[str], str, None] = None,
55    _patch_args: Optional[Dict[str, Any]] = None,
56    _use_cli_daemon: bool = True,
57    _session_id: Optional[str] = None,
58) -> SuccessTuple:
59    """
60    Parse arguments and launch a Meerschaum action.
61
62    Returns
63    -------
64    A `SuccessTuple` indicating success.
65    """
66    start = time.perf_counter()
67    from meerschaum.config.environment import get_daemon_env_vars
68    sysargs_list = shlex.split(sysargs) if isinstance(sysargs, str) else sysargs
69    if (
70        not _use_cli_daemon
71        or (not sysargs or (sysargs[0] and sysargs[0].startswith('-')))
72        or '--no-daemon' in sysargs_list
73        or '--daemon' in sysargs_list
74        or '-d' in sysargs_list
75        or get_daemon_env_vars()
76        or not mrsm.get_config('system', 'experimental', 'cli_daemon')
77    ):
78        success, msg = entry_without_daemon(sysargs, _patch_args=_patch_args)
79        end = time.perf_counter()
80        if '--debug' in sysargs_list:
81            print(f"Duration without daemon: {round(end - start, 3)}")
82        return success, msg
83
84    from meerschaum._internal.cli.entry import entry_with_daemon
85    success, msg = entry_with_daemon(sysargs, _patch_args=_patch_args)
86    end = time.perf_counter()
87    if '--debug' in sysargs_list:
88        print(f"Duration with daemon: {round(end - start, 3)}")
89    return success, msg

Parse arguments and launch a Meerschaum action.

Returns