meerschaum

Meerschaum Python API
Welcome to the Meerschaum Python API technical documentation! Here you can find information about the classes and functions provided by the meerschaum package. Visit meerschaum.io for general usage documentation.
Root Module
For your convenience, the following classes and functions may be imported from the root meerschaum namespace:
Classes
Examples
Build a Connector
Get existing connectors or build a new one in-memory with the meerschaum.get_connector() factory function:
import meerschaum as mrsm
sql_conn = mrsm.get_connector(
'sql:temp',
flavor='sqlite',
database='/tmp/tmp.db',
)
df = sql_conn.read("SELECT 1 AS foo")
print(df)
# foo
# 0 1
sql_conn.to_sql(df, 'foo')
print(sql_conn.read('foo'))
# foo
# 0 1
Create a Custom Connector Class
Decorate your connector classes with meerschaum.make_connector() to designate it as a custom connector:
from datetime import datetime, timezone
from random import randint
import meerschaum as mrsm
from meerschaum.utils.dtypes import round_time
@mrsm.make_connector
class FooConnector(mrsm.Connector):
REQUIRED_ATTRIBUTES = ['username', 'password']
def fetch(
self,
begin: datetime | None = None,
end: datetime | None = None,
):
now = begin or round_time(datetime.now(timezone.utc))
return [
{'ts': now, 'id': 1, 'vl': randint(1, 100)},
{'ts': now, 'id': 2, 'vl': randint(1, 100)},
{'ts': now, 'id': 3, 'vl': randint(1, 100)},
]
foo_conn = mrsm.get_connector(
'foo:bar',
username='foo',
password='bar',
)
docs = foo_conn.fetch()
Build a Pipe
Build a meerschaum.Pipe in-memory:
from datetime import datetime
import meerschaum as mrsm
pipe = mrsm.Pipe(
foo_conn, 'demo',
instance=sql_conn,
columns={'datetime': 'ts', 'id': 'id'},
tags=['production'],
)
pipe.sync(begin=datetime(2024, 1, 1))
df = pipe.get_data()
print(df)
# ts id vl
# 0 2024-01-01 1 97
# 1 2024-01-01 2 18
# 2 2024-01-01 3 96
Add temporary=True to skip registering the pipe in the pipes table.
Query an Integer-Axis Pipe by Datetime
When a pipe's datetime axis is an integer epoch, set precision so datetime bounds can be translated to the axis's integer value. This lets you pass a datetime begin / end to meerschaum.Pipe.get_data() (and to actions like show data, clear, and deduplicate):
from datetime import datetime, timezone
import meerschaum as mrsm
pipe = mrsm.Pipe(
'demo', 'epoch',
instance='sql:temp',
columns={'datetime': 'ts'},
dtypes={'ts': 'int'},
precision='millisecond',
)
pipe.sync([{'ts': 1780099200000}])
### The datetime is translated to the epoch value `1780099200000`.
df = pipe.get_data(begin='2026-05-30')
Integer bounds (begin=1780099200000) still pass through unchanged. A datetime bound on a non-epoch integer axis (no precision set) raises a ValueError. Convert directly with meerschaum.utils.dtypes.datetime_to_int():
from datetime import datetime, timezone
from meerschaum.utils.dtypes import datetime_to_int
datetime_to_int(datetime(2026, 5, 30, tzinfo=timezone.utc), 'millisecond')
# 1780099200000
Get Registered Pipes
The meerschaum.get_pipes() function returns a dictionary hierarchy of pipes by connector, metric, and location:
import meerschaum as mrsm
pipes = mrsm.get_pipes(instance='sql:temp')
pipe = pipes['foo:bar']['demo'][None]
Add as_list=True to flatten the hierarchy:
import meerschaum as mrsm
pipes = mrsm.get_pipes(
tags=['production'],
instance=sql_conn,
as_list=True,
)
print(pipes)
# [Pipe('foo:bar', 'demo', instance='sql:temp')]
Filter by the dtype of the datetime index column with datetime_dtypes. Accepted values are 'datetime', 'int', and 'None'; prefix with '_' to negate:
import meerschaum as mrsm
### Only pipes with a timestamp datetime index:
timestamp_pipes = mrsm.get_pipes(datetime_dtypes=['datetime'], as_list=True)
### Only pipes with an integer datetime index:
int_pipes = mrsm.get_pipes(datetime_dtypes=['int'], as_list=True)
### Exclude pipes without a datetime index:
datetime_pipes = mrsm.get_pipes(datetime_dtypes=['_None'], as_list=True)
Import Plugins
You can import a plugin's module through meerschaum.Plugin.module:
import meerschaum as mrsm
plugin = mrsm.Plugin('noaa')
with mrsm.Venv(plugin):
noaa = plugin.module
If your plugin has submodules, use meerschaum.plugins.from_plugin_import:
from meerschaum.plugins import from_plugin_import
get_defined_pipes = from_plugin_import('compose.utils.pipes', 'get_defined_pipes')
Import multiple plugins with meerschaum.plugins.import_plugins:
from meerschaum.plugins import import_plugins
noaa, compose = import_plugins('noaa', 'compose')
Create a Job
Create a meerschaum.Job with name and sysargs:
import meerschaum as mrsm
job = mrsm.Job('syncing-engine', 'sync pipes --loop')
success, msg = job.start()
Pass executor_keys as the connectors keys of an API instance to create a remote job:
import meerschaum as mrsm
job = mrsm.Job(
'foo',
'sync pipes -s daily',
executor_keys='api:main',
)
Import from a Virtual Environment
Use the meerschaum.Venv context manager to activate a virtual environment:
import meerschaum as mrsm
with mrsm.Venv('noaa'):
import requests
print(requests.__file__)
# /home/bmeares/.config/meerschaum/venvs/noaa/lib/python3.12/site-packages/requests/__init__.py
To import packages which may not be installed, use meerschaum.attempt_import():
import meerschaum as mrsm
requests = mrsm.attempt_import('requests', venv='noaa')
print(requests.__file__)
# /home/bmeares/.config/meerschaum/venvs/noaa/lib/python3.12/site-packages/requests/__init__.py
Run Actions
Run sysargs with meerschaum.entry():
import meerschaum as mrsm
success, msg = mrsm.entry('show pipes + show version : x2')
Use meerschaum.actions.get_action() to access an action function directly:
from meerschaum.actions import get_action
show_pipes = get_action(['show', 'pipes'])
success, msg = show_pipes(connector_keys=['plugin:noaa'])
Get a dictionary of available subactions with meerschaum.actions.get_subactions():
from meerschaum.actions import get_subactions
subactions = get_subactions('show')
success, msg = subactions['pipes']()
Create a Plugin
Run bootstrap plugin to create a new plugin:
mrsm bootstrap plugin example
This will create example.py in your plugins directory (default ~/.config/meerschaum/plugins/, Windows: %APPDATA%\Meerschaum\plugins). You may paste the example code from the "Create a Custom Action" example below.
Open your plugin with edit plugin:
mrsm edit plugin example
Run edit plugin and paste the example code below to try out the features.
See the writing plugins guide for more in-depth documentation.
Create a Custom Action
Decorate a function with meerschaum.actions.make_action to designate it as an action. Subactions will be automatically detected if not decorated:
from meerschaum.actions import make_action
@make_action
def sing():
print('What would you like me to sing?')
return True, "Success"
def sing_tune():
return False, "I don't know that song!"
def sing_song():
print('Hello, World!')
return True, "Success"
Use meerschaum.plugins.add_plugin_argument() to create new parameters for your action:
from meerschaum.plugins import make_action, add_plugin_argument
add_plugin_argument(
'--song', type=str, help='What song to sing.',
)
@make_action
def sing_melody(action=None, song=None):
to_sing = action[0] if action else song
if not to_sing:
return False, "Please tell me what to sing!"
return True, f'~I am singing {to_sing}~'
mrsm sing melody lalala
mrsm sing melody --song do-re-mi
Add a Page to the Web Dashboard
Use the decorators meerschaum.plugins.dash_plugin() and meerschaum.plugins.web_page() to add new pages to the web dashboard:
from meerschaum.plugins import dash_plugin, web_page
@dash_plugin
def init_dash(dash_app):
import dash.html as html
import dash_bootstrap_components as dbc
from dash import Input, Output, no_update
### Routes to '/dash/my-page'
@web_page('/my-page', login_required=False)
def my_page():
return dbc.Container([
html.H1("Hello, World!"),
dbc.Button("Click me", id='my-button'),
html.Div(id="my-output-div"),
])
@dash_app.callback(
Output('my-output-div', 'children'),
Input('my-button', 'n_clicks'),
)
def my_button_click(n_clicks):
if not n_clicks:
return no_update
return html.P(f'You clicked {n_clicks} times!')
Submodules
meerschaum.actions
Access functions for actions and subactions.
meerschaum.actions.actionsmeerschaum.actions.get_action()meerschaum.actions.get_completer()meerschaum.actions.get_main_action_name()meerschaum.actions.get_subactions()
meerschaum.config
Read and write the Meerschaum configuration registry.
meerschaum.config.get_config()meerschaum.config.get_plugin_config()meerschaum.config.write_config()meerschaum.config.write_plugin_config()meerschaum.config.environment
Patch configuration and connectors from environment variables.
meerschaum.config.environment.apply_environment_patches()meerschaum.config.environment.apply_environment_config()meerschaum.config.environment.apply_environment_uris()meerschaum.config.environment.apply_connector_uri()meerschaum.config.environment.get_connector_env_regex()meerschaum.config.environment.get_connector_env_vars()meerschaum.config.environment.get_env_vars()meerschaum.config.environment.get_daemon_env_vars()meerschaum.config.environment.replace_env()
meerschaum.connectors
Build connectors to interact with databases and fetch data.
meerschaum.connectors.get_connector()meerschaum.connectors.make_connector()meerschaum.connectors.is_connected()meerschaum.connectors.poll.retry_connect()meerschaum.connectors.Connectormeerschaum.connectors.sql.SQLConnectormeerschaum.connectors.api.APIConnectormeerschaum.connectors.valkey.ValkeyConnector
meerschaum.jobs
Start background jobs.
meerschaum.jobs.Jobmeerschaum.jobs.Executormeerschaum.jobs.systemd.SystemdExecutormeerschaum.jobs.get_jobs()meerschaum.jobs.get_filtered_jobs()meerschaum.jobs.get_running_jobs()meerschaum.jobs.get_stopped_jobs()meerschaum.jobs.get_paused_jobs()meerschaum.jobs.get_restart_jobs()meerschaum.jobs.make_executor()meerschaum.jobs.check_restart_jobs()meerschaum.jobs.start_check_jobs_thread()meerschaum.jobs.stop_check_jobs_thread()meerschaum.jobs.get_executor_keys_from_context()
meerschaum.plugins
Access plugin modules and other API utilties.
meerschaum.plugins.Pluginmeerschaum.plugins.api_plugin()meerschaum.plugins.dash_plugin()meerschaum.plugins.import_plugins()meerschaum.plugins.reload_plugins()meerschaum.plugins.get_plugins()meerschaum.plugins.get_data_plugins()meerschaum.plugins.add_plugin_argument()meerschaum.plugins.pre_sync_hook()meerschaum.plugins.post_sync_hook()
meerschaum.utils
Utility functions are available in several submodules:
meerschaum.utils.daemon.daemon_entry()meerschaum.utils.daemon.daemon_action()meerschaum.utils.daemon.get_daemons()meerschaum.utils.daemon.get_daemon_ids()meerschaum.utils.daemon.get_running_daemons()meerschaum.utils.daemon.get_paused_daemons()meerschaum.utils.daemon.get_stopped_daemons()meerschaum.utils.daemon.get_filtered_daemons()meerschaum.utils.daemon.run_daemon()meerschaum.utils.daemon.Daemonmeerschaum.utils.daemon.FileDescriptorInterceptormeerschaum.utils.daemon.RotatingFile
meerschaum.utils.daemon
Manage background jobs.
meerschaum.utils.dataframe.add_missing_cols_to_df()meerschaum.utils.dataframe.chunksize_to_npartitions()meerschaum.utils.dataframe.df_from_literal()meerschaum.utils.dataframe.df_is_chunk_generator()meerschaum.utils.dataframe.enforce_dtypes()meerschaum.utils.dataframe.filter_unseen_df()meerschaum.utils.dataframe.get_bool_cols()meerschaum.utils.dataframe.get_bytes_cols()meerschaum.utils.dataframe.get_datetime_bound_from_df()meerschaum.utils.dataframe.get_date_cols()meerschaum.utils.dataframe.get_datetime_cols()meerschaum.utils.dataframe.get_datetime_cols_types()meerschaum.utils.dataframe.get_first_valid_dask_partition()meerschaum.utils.dataframe.get_geometry_cols()meerschaum.utils.dataframe.get_geometry_cols_types()meerschaum.utils.dataframe.get_json_cols()meerschaum.utils.dataframe.get_numeric_cols()meerschaum.utils.dataframe.get_special_cols()meerschaum.utils.dataframe.get_unhashable_cols()meerschaum.utils.dataframe.get_unique_index_values()meerschaum.utils.dataframe.get_uuid_cols()meerschaum.utils.dataframe.parse_df_datetimes()meerschaum.utils.dataframe.query_df()meerschaum.utils.dataframe.to_json()meerschaum.utils.dataframe.to_simple_lines()meerschaum.utils.dataframe.parse_simple_lines()
meerschaum.utils.dataframe
Manipulate dataframes.
meerschaum.utils.dtypes.are_dtypes_equal()meerschaum.utils.dtypes.attempt_cast_to_bytes()meerschaum.utils.dtypes.attempt_cast_to_geometry()meerschaum.utils.dtypes.attempt_cast_to_numeric()meerschaum.utils.dtypes.attempt_cast_to_uuid()meerschaum.utils.dtypes.coerce_timezone()meerschaum.utils.dtypes.datetime_to_int()meerschaum.utils.dtypes.deserialize_base64()meerschaum.utils.dtypes.deserialize_bytes_string()meerschaum.utils.dtypes.deserialize_geometry()meerschaum.utils.dtypes.encode_bytes_for_bytea()meerschaum.utils.dtypes.geometry_is_gpkg()meerschaum.utils.dtypes.geometry_is_wkt()meerschaum.utils.dtypes.get_current_timestamp()meerschaum.utils.dtypes.get_geometry_type_srid()meerschaum.utils.dtypes.is_dtype_numeric()meerschaum.utils.dtypes.is_dtype_special()meerschaum.utils.dtypes.json_serialize_value()meerschaum.utils.dtypes.none_if_null()meerschaum.utils.dtypes.project_geometry()meerschaum.utils.dtypes.quantize_decimal()meerschaum.utils.dtypes.serialize_bytes()meerschaum.utils.dtypes.serialize_datetime()meerschaum.utils.dtypes.serialize_date()meerschaum.utils.dtypes.serialize_decimal()meerschaum.utils.dtypes.serialize_geometry()meerschaum.utils.dtypes.to_datetime()meerschaum.utils.dtypes.to_pandas_dtype()meerschaum.utils.dtypes.value_is_null()meerschaum.utils.dtypes.get_next_precision_unit()meerschaum.utils.dtypes.round_time()
meerschaum.utils.dtypes
Work with data types.
meerschaum.utils.formatting.colored()meerschaum.utils.formatting.extract_stats_from_message()meerschaum.utils.formatting.fill_ansi()meerschaum.utils.formatting.format_bytes()meerschaum.utils.formatting.format_dataframe()meerschaum.utils.formatting.get_console()meerschaum.utils.formatting.highlight_pipes()meerschaum.utils.formatting.make_header()meerschaum.utils.formatting.pipe_repr()meerschaum.utils.formatting.pprint()meerschaum.utils.formatting.pprint_df()meerschaum.utils.formatting.pprint_pipes()meerschaum.utils.formatting.print_options()meerschaum.utils.formatting.print_pipes_results()meerschaum.utils.formatting.print_tuple()meerschaum.utils.formatting.translate_rich_to_termcolor()
meerschaum.utils.formatting
Format output text.
meerschaum.utils.misc.items_str()meerschaum.utils.misc.is_int()meerschaum.utils.misc.is_uuid()meerschaum.utils.misc.interval_str()meerschaum.utils.misc.filter_keywords()meerschaum.utils.misc.generate_password()meerschaum.utils.misc.string_to_dict()meerschaum.utils.misc.iterate_chunks()meerschaum.utils.misc.timed_input()meerschaum.utils.misc.replace_pipes_in_dict()meerschaum.utils.misc.is_valid_email()meerschaum.utils.misc.string_width()meerschaum.utils.misc.replace_password()meerschaum.utils.misc.parse_config_substitution()meerschaum.utils.misc.edit_file()meerschaum.utils.misc.get_in_ex_params()meerschaum.utils.misc.separate_negation_values()meerschaum.utils.misc.flatten_list()meerschaum.utils.misc.make_symlink()meerschaum.utils.misc.is_symlink()meerschaum.utils.misc.wget()meerschaum.utils.misc.add_method_to_class()meerschaum.utils.misc.is_pipe_registered()meerschaum.utils.misc.get_cols_lines()meerschaum.utils.misc.sorted_dict()meerschaum.utils.misc.flatten_pipes_dict()meerschaum.utils.misc.dict_from_od()meerschaum.utils.misc.remove_ansi()meerschaum.utils.misc.get_connector_labels()meerschaum.utils.misc.json_serialize_datetime()meerschaum.utils.misc.async_wrap()meerschaum.utils.misc.is_docker_available()meerschaum.utils.misc.is_android()meerschaum.utils.misc.is_bcp_available()meerschaum.utils.misc.truncate_string_sections()meerschaum.utils.misc.safely_extract_tar()meerschaum.utils.misc.get_directory_size()
meerschaum.utils.misc
Miscellaneous utility functions.
meerschaum.utils.packages.attempt_import()meerschaum.utils.packages.get_module_path()meerschaum.utils.packages.manually_import_module()meerschaum.utils.packages.get_install_no_version()meerschaum.utils.packages.determine_version()meerschaum.utils.packages.need_update()meerschaum.utils.packages.get_pip()meerschaum.utils.packages.pip_install()meerschaum.utils.packages.pip_uninstall()meerschaum.utils.packages.completely_uninstall_package()meerschaum.utils.packages.run_python_package()meerschaum.utils.packages.lazy_import()meerschaum.utils.packages.pandas_name()meerschaum.utils.packages.import_pandas()meerschaum.utils.packages.import_rich()meerschaum.utils.packages.import_dcc()meerschaum.utils.packages.import_html()meerschaum.utils.packages.get_modules_from_package()meerschaum.utils.packages.import_children()meerschaum.utils.packages.reload_package()meerschaum.utils.packages.reload_meerschaum()meerschaum.utils.packages.is_installed()meerschaum.utils.packages.venv_contains_package()meerschaum.utils.packages.package_venv()meerschaum.utils.packages.ensure_readline()meerschaum.utils.packages.get_prerelease_dependencies()
meerschaum.utils.packages
Manage Python packages.
meerschaum.utils.pipes
Utilities for working with pipe objects.
meerschaum.utils.sql.build_where()meerschaum.utils.sql.clean()meerschaum.utils.sql.get_sqlalchemy_table()meerschaum.utils.sql.dateadd_str()meerschaum.utils.sql.test_connection()meerschaum.utils.sql.get_distinct_col_count()meerschaum.utils.sql.sql_item_name()meerschaum.utils.sql.pg_capital()meerschaum.utils.sql.oracle_capital()meerschaum.utils.sql.truncate_item_name()meerschaum.utils.sql.table_exists()meerschaum.utils.sql.get_table_cols_types()meerschaum.utils.sql.get_table_cols_indices()meerschaum.utils.sql.get_update_queries()meerschaum.utils.sql.get_null_replacement()meerschaum.utils.sql.get_db_version()meerschaum.utils.sql.get_rename_table_queries()meerschaum.utils.sql.get_create_table_queries()meerschaum.utils.sql.wrap_query_with_cte()meerschaum.utils.sql.format_cte_subquery()meerschaum.utils.sql.session_execute()meerschaum.utils.sql.get_reset_autoincrement_queries()meerschaum.utils.sql.get_postgis_geo_columns_types()meerschaum.utils.sql.get_create_schema_if_not_exists_queries()
meerschaum.utils.sql
Build SQL queries.
meerschaum.utils.threading
Manage threads and process-wide stop signals.
meerschaum.utils.venv.Venvmeerschaum.utils.venv.activate_venv()meerschaum.utils.venv.deactivate_venv()meerschaum.utils.venv.get_module_venv()meerschaum.utils.venv.get_venvs()meerschaum.utils.venv.init_venv()meerschaum.utils.venv.inside_venv()meerschaum.utils.venv.is_venv_active()meerschaum.utils.venv.venv_exec()meerschaum.utils.venv.venv_executable()meerschaum.utils.venv.venv_exists()meerschaum.utils.venv.venv_target_path()meerschaum.utils.venv.verify_venv()
meerschaum.utils.venv
Manage virtual environments.
meerschaum.utils.warnings
Print warnings, errors, info, and debug messages.
1#! /usr/bin/env python 2# -*- coding: utf-8 -*- 3# vim:fenc=utf-8 4 5""" 6Copyright 2020–2026 Bennett Meares 7 8Licensed under the Apache License, Version 2.0 (the "License"); 9you may not use this file except in compliance with the License. 10You may obtain a copy of the License at 11 12 http://www.apache.org/licenses/LICENSE-2.0 13 14Unless required by applicable law or agreed to in writing, software 15distributed under the License is distributed on an "AS IS" BASIS, 16WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17See the License for the specific language governing permissions and 18limitations under the License. 19""" 20 21import atexit 22 23from meerschaum.utils.typing import SuccessTuple 24from meerschaum.utils.packages import attempt_import 25from meerschaum.core.Pipe import Pipe 26from meerschaum.plugins import Plugin 27from meerschaum.utils.venv import Venv 28from meerschaum.jobs import Job, make_executor 29from meerschaum.connectors import get_connector, Connector, InstanceConnector, make_connector 30from meerschaum.utils import get_pipes 31from meerschaum.utils.formatting import pprint 32from meerschaum._internal.docs import index as __doc__ 33from meerschaum.config import __version__, get_config 34from meerschaum._internal.entry import entry 35from meerschaum.__main__ import _close_pools 36 37atexit.register(_close_pools) 38 39__pdoc__ = {'gui': False, 'api': False, 'core': False, '_internal': False} 40__all__ = ( 41 "get_pipes", 42 "get_connector", 43 "get_config", 44 "Pipe", 45 "Plugin", 46 "SuccessTuple", 47 "Venv", 48 "Plugin", 49 "Job", 50 "pprint", 51 "attempt_import", 52 "actions", 53 "config", 54 "connectors", 55 "jobs", 56 "plugins", 57 "utils", 58 "SuccessTuple", 59 "Connector", 60 "InstanceConnector", 61 "make_connector", 62 "entry", 63)
29def get_pipes( 30 connector_keys: Union[str, List[str], None] = None, 31 metric_keys: Union[str, List[str], None] = None, 32 location_keys: Union[str, List[str], None] = None, 33 tags: Optional[List[str]] = None, 34 targets: Optional[List[str]] = None, 35 datetime_dtypes: Optional[List[str]] = None, 36 params: Optional[Dict[str, Any]] = None, 37 mrsm_instance: Union[str, InstanceConnector, None] = None, 38 instance: Union[str, InstanceConnector, None] = None, 39 as_list: bool = False, 40 as_tags_dict: bool = False, 41 as_targets_dict: bool = False, 42 method: str = 'registered', 43 workers: Optional[int] = None, 44 debug: bool = False, 45 _cache_parameters: bool = True, 46 **kw: Any 47) -> Union[PipesDict, List[mrsm.Pipe], Dict[str, mrsm.Pipe]]: 48 """ 49 Return a dictionary or list of `meerschaum.Pipe` objects. 50 51 Parameters 52 ---------- 53 connector_keys: Union[str, List[str], None], default None 54 String or list of connector keys. 55 If omitted or is `'*'`, fetch all possible keys. 56 If a string begins with `'_'`, select keys that do NOT match the string. 57 58 metric_keys: Union[str, List[str], None], default None 59 String or list of metric keys. See `connector_keys` for formatting. 60 61 location_keys: Union[str, List[str], None], default None 62 String or list of location keys. See `connector_keys` for formatting. 63 64 tags: Optional[List[str]], default None 65 If provided, only include pipes with these tags. 66 67 datetime_dtypes: Optional[List[str]], default None 68 If provided, only include pipes with the corresponding `datetime` axis dtypes. 69 Accepted values are `datetime`, `int`, `None` (or `null`, etc.). 70 May be negated by `_`. 71 72 params: Optional[Dict[str, Any]], default None 73 Dictionary of additional parameters to search by. 74 Params are parsed into a SQL WHERE clause. 75 E.g. `{'a': 1, 'b': 2}` equates to `'WHERE a = 1 AND b = 2'` 76 77 mrsm_instance: Union[str, InstanceConnector, None], default None 78 Connector keys for the Meerschaum instance of the pipes. 79 Must be a `meerschaum.connectors.sql.SQLConnector.SQLConnector` or 80 `meerschaum.connectors.api.APIConnector.APIConnector`. 81 82 as_list: bool, default False 83 If `True`, return pipes in a list instead of a hierarchical dictionary. 84 `False` : `{connector_keys: {metric_key: {location_key: Pipe}}}` 85 `True` : `[Pipe]` 86 87 as_tags_dict: bool, default False 88 If `True`, return a dictionary mapping tags to pipes. 89 Pipes with multiple tags will be repeated. 90 91 as_targets_dict: bool, default False 92 If `True`, return a dictionary mapping `(schema, target)` tuples to pipes. 93 Pipes sharing the same target across different schemata are grouped separately. 94 95 method: str, default 'registered' 96 Available options: `['registered', 'explicit', 'all']` 97 If `'registered'` (default), create pipes based on registered keys in the connector's pipes table 98 (API or SQL connector, depends on mrsm_instance). 99 If `'explicit'`, create pipes from provided connector_keys, metric_keys, and location_keys 100 instead of consulting the pipes table. Useful for creating non-existent pipes. 101 If `'all'`, create pipes from predefined metrics and locations. Required `connector_keys`. 102 **NOTE:** Method `'all'` is not implemented! 103 104 workers: Optional[int], default None 105 If provided (and `as_tags_dict` or `as_targets_dict` is `True`), set the number of workers 106 for the pool to fetch tags or targets. 107 Only takes effect if the instance connector supports multi-threading. 108 109 **kw: Any: 110 Keyword arguments to pass to the `meerschaum.Pipe` constructor. 111 112 Returns 113 ------- 114 A dictionary of dictionaries and `meerschaum.Pipe` objects 115 in the connector, metric, location hierarchy. 116 If `as_list` is `True`, return a list of `meerschaum.Pipe` objects. 117 If `as_tags_dict` is `True`, return a dictionary mapping tags to pipes. 118 If `as_targets_dict` is `True`, return a dictionary mapping targets to pipes. 119 120 Examples 121 -------- 122 ``` 123 >>> ### Manual definition: 124 >>> pipes = { 125 ... <connector_keys>: { 126 ... <metric_key>: { 127 ... <location_key>: Pipe( 128 ... <connector_keys>, 129 ... <metric_key>, 130 ... <location_key>, 131 ... ), 132 ... }, 133 ... }, 134 ... }, 135 >>> ### Accessing a single pipe: 136 >>> pipes['sql:main']['weather'][None] 137 >>> ### Return a list instead: 138 >>> get_pipes(as_list=True) 139 [Pipe('sql:main', 'weather')] 140 >>> get_pipes(as_tags_dict=True) 141 {'gvl': Pipe('sql:main', 'weather')} 142 ``` 143 """ 144 import json 145 from collections import defaultdict 146 from meerschaum.config import get_config 147 from meerschaum.config.static import STATIC_CONFIG 148 from meerschaum.utils.warnings import error 149 from meerschaum.utils.misc import filter_keywords, separate_negation_values 150 from meerschaum.utils.pool import get_pool 151 from meerschaum.utils.pipes import replace_pipes_syntax 152 from meerschaum.utils.debug import dprint 153 from meerschaum.utils.dtypes import value_is_null, get_current_timestamp 154 from meerschaum import Pipe 155 156 negation_prefix = STATIC_CONFIG['system']['fetch_pipes_keys']['negation_prefix'] 157 if datetime_dtypes: 158 if isinstance(datetime_dtypes, str): 159 datetime_dtypes = [datetime_dtypes] 160 for _dt in datetime_dtypes: 161 _clean = str(_dt).lstrip(negation_prefix).lower() 162 if _clean not in ('datetime', 'int') and not value_is_null(_clean): 163 error(f"Invalid datetime dtype '{_dt}'.") 164 165 if connector_keys is None: 166 connector_keys = [] 167 if metric_keys is None: 168 metric_keys = [] 169 if location_keys is None: 170 location_keys = [] 171 if params is None: 172 params = {} 173 if tags is None: 174 tags = [] 175 176 if isinstance(connector_keys, str): 177 connector_keys = [connector_keys] 178 if isinstance(metric_keys, str): 179 metric_keys = [metric_keys] 180 if isinstance(location_keys, str): 181 location_keys = [location_keys] 182 183 ### Get SQL or API connector (keys come from `connector.fetch_pipes_keys()`). 184 if mrsm_instance is None: 185 mrsm_instance = instance 186 if mrsm_instance is None: 187 mrsm_instance = get_config('meerschaum', 'instance', patch=True) 188 if isinstance(mrsm_instance, str): 189 from meerschaum.connectors.parse import parse_instance_keys 190 connector = parse_instance_keys(keys=mrsm_instance, debug=debug) 191 else: 192 from meerschaum.connectors import instance_types 193 valid_connector = False 194 if hasattr(mrsm_instance, 'type'): 195 if mrsm_instance.type in instance_types: 196 valid_connector = True 197 if not valid_connector: 198 error(f"Invalid instance connector: {mrsm_instance}") 199 connector = mrsm_instance 200 if debug: 201 dprint(f"Using instance connector: {connector}") 202 if not connector: 203 error(f"Could not create connector from keys: '{mrsm_instance}'") 204 205 ### Get a list of tuples for the keys needed to build pipes. 206 result = fetch_pipes_keys( 207 method, 208 connector, 209 connector_keys = connector_keys, 210 metric_keys = metric_keys, 211 location_keys = location_keys, 212 tags = tags, 213 params = params, 214 workers = workers, 215 debug = debug 216 ) 217 if result is None: 218 error("Unable to build pipes!") 219 result_items: List[Tuple] = ( 220 list(result.items()) 221 if isinstance(result, dict) 222 else [(None, keys_tuple) for keys_tuple in result] 223 ) 224 225 ### Populate the `pipes` dictionary with Pipes based on the keys 226 ### obtained from the chosen `method`. 227 in_dtypes, ex_dtypes = separate_negation_values(datetime_dtypes or []) 228 in_targets, ex_targets = separate_negation_values(targets or []) 229 pipes: PipesDict = {} 230 targets_pipes: Dict[Tuple[Optional[str], str], List[mrsm.Pipe]] = defaultdict(lambda: []) 231 connector_schema = getattr(connector, 'schema', None) 232 connector_is_sql = getattr(connector, 'type', None) == 'sql' 233 connector_flavor = getattr(connector, 'flavor', None) 234 for pipe_id, keys_tuple in result_items: 235 ck, mk, lk = keys_tuple[0], keys_tuple[1], keys_tuple[2] 236 pipe_tags_or_parameters = keys_tuple[3] if len(keys_tuple) == 4 else None 237 pipe_parameters = ( 238 pipe_tags_or_parameters 239 if isinstance(pipe_tags_or_parameters, (dict, str)) 240 else None 241 ) 242 if isinstance(pipe_parameters, str): 243 pipe_parameters = json.loads(pipe_parameters) 244 pipe_tags = ( 245 pipe_tags_or_parameters 246 if isinstance(pipe_tags_or_parameters, list) 247 else ( 248 pipe_tags_or_parameters.get('tags', []) 249 if isinstance(pipe_tags_or_parameters, dict) 250 else None 251 ) 252 ) 253 254 pipe = Pipe( 255 ck, mk, lk, 256 mrsm_instance = connector, 257 parameters = pipe_parameters, 258 tags = pipe_tags, 259 debug = debug, 260 **filter_keywords(Pipe, **kw) 261 ) 262 pipe.__dict__['_tags'] = pipe_tags 263 if pipe_id is not None: 264 pipe._cache_value('_id', pipe_id, memory_only=True, debug=debug) 265 if pipe_parameters is not None: 266 now = get_current_timestamp('ms', as_int=True) / 1000 267 full_attributes = { 268 'connector_keys': ck, 269 'metric_key': mk, 270 'location_key': lk, 271 'parameters': pipe_parameters, 272 } 273 if pipe_id is not None: 274 full_attributes['pipe_id'] = pipe_id 275 pipe._cache_value('attributes', full_attributes, memory_only=True, debug=debug) 276 pipe._cache_value('_attributes_sync_time', now, memory_only=True, debug=debug) 277 278 if datetime_dtypes or targets: 279 parameters_str = str(pipe_parameters) 280 if pipe_parameters is None or 'MRSM{' in parameters_str or 'Pipe(' in parameters_str: 281 pipe_parameters = pipe.get_parameters(debug=debug) 282 283 keep_pipe = True 284 285 if datetime_dtypes: 286 columns_val = (pipe_parameters or {}).get('columns', {}) or {} 287 dt_col = columns_val.get('datetime', None) 288 pipe_dtypes = ( 289 ((pipe_parameters or {}).get('dtypes', None) or {}) 290 if dt_col 291 else None 292 ) 293 dt_typ = pipe_dtypes.get(dt_col, None) if dt_col else None 294 295 def _dtype_matches(clean_d): 296 if not dt_col: 297 return value_is_null(clean_d) 298 return ( 299 (clean_d == 'int' and 'int' in str(dt_typ).lower()) 300 or 301 (clean_d == 'datetime' and 'int' not in str(dt_typ).lower()) 302 ) 303 304 in_match = not in_dtypes or any(_dtype_matches(d) for d in in_dtypes) 305 ex_match = bool(ex_dtypes and any(_dtype_matches(d) for d in ex_dtypes)) 306 keep_pipe = keep_pipe and in_match and not ex_match 307 if not keep_pipe: 308 continue 309 310 if targets: 311 pipe_target = pipe.target 312 in_target_match = not in_targets or any(t == pipe_target for t in in_targets) 313 ex_target_match = bool(ex_targets and any(t == pipe_target for t in ex_targets)) 314 keep_pipe = keep_pipe and in_target_match and not ex_target_match 315 if not keep_pipe: 316 continue 317 318 if ck not in pipes: 319 pipes[ck] = {} 320 321 if mk not in pipes[ck]: 322 pipes[ck][mk] = {} 323 324 325 pipes[ck][mk][lk] = pipe 326 327 if as_targets_dict: 328 raw_params = pipe_parameters if isinstance(pipe_parameters, dict) else {} 329 schema = raw_params.get('schema') or connector_schema 330 explicit_target = ( 331 raw_params.get('target') 332 or raw_params.get('target_name') 333 or raw_params.get('target_table') 334 or raw_params.get('target_table_name') 335 ) 336 if explicit_target: 337 target_name = ( 338 replace_pipes_syntax(explicit_target, _pipe=pipe) 339 if isinstance(explicit_target, str) and '{{' in explicit_target 340 else explicit_target 341 ) 342 else: 343 target_name = pipe._target_legacy() 344 if connector_is_sql and connector_flavor: 345 from meerschaum.utils.sql import truncate_item_name 346 target_name = truncate_item_name(target_name, connector_flavor) 347 targets_pipes[(schema, target_name)].append(pipe) 348 349 if not as_list and not as_tags_dict and not as_targets_dict: 350 return pipes 351 352 from meerschaum.utils.pipes import flatten_pipes_dict 353 pipes_list = flatten_pipes_dict(pipes) 354 if as_list: 355 return pipes_list 356 357 pool = get_pool(workers=(workers if connector.IS_THREAD_SAFE else 1)) 358 359 def gather_pipe_tags(pipe: mrsm.Pipe) -> Tuple[mrsm.Pipe, List[str]]: 360 _tags = pipe.__dict__.get('_tags', None) 361 gathered_tags = _tags if _tags is not None else pipe.tags 362 return pipe, (gathered_tags or []) 363 364 if as_tags_dict: 365 tags_pipes = defaultdict(lambda: []) 366 pipes_tags = dict(pool.map(gather_pipe_tags, pipes_list)) 367 for pipe, tags in pipes_tags.items(): 368 for tag in (tags or []): 369 tags_pipes[tag].append(pipe) 370 371 return dict(tags_pipes) 372 373 if as_targets_dict: 374 return dict(targets_pipes) 375 376 raise NotImplementedError("No futher options for returning pipes.")
Return a dictionary or list of meerschaum.Pipe objects.
Parameters
- connector_keys (Union[str, List[str], None], default None):
String or list of connector keys.
If omitted or is
'*', fetch all possible keys. If a string begins with'_', select keys that do NOT match the string. - metric_keys (Union[str, List[str], None], default None):
String or list of metric keys. See
connector_keysfor formatting. - location_keys (Union[str, List[str], None], default None):
String or list of location keys. See
connector_keysfor formatting. - tags (Optional[List[str]], default None): If provided, only include pipes with these tags.
- datetime_dtypes (Optional[List[str]], default None):
If provided, only include pipes with the corresponding
datetimeaxis dtypes. Accepted values aredatetime,int,None(ornull, etc.). May be negated by_. - params (Optional[Dict[str, Any]], default None):
Dictionary of additional parameters to search by.
Params are parsed into a SQL WHERE clause.
E.g.
{'a': 1, 'b': 2}equates to'WHERE a = 1 AND b = 2' - mrsm_instance (Union[str, InstanceConnector, None], default None):
Connector keys for the Meerschaum instance of the pipes.
Must be a
meerschaum.connectors.sql.SQLConnector.SQLConnectorormeerschaum.connectors.api.APIConnector.APIConnector. - as_list (bool, default False):
If
True, return pipes in a list instead of a hierarchical dictionary.False:{connector_keys: {metric_key: {location_key: Pipe}}}True:[Pipe] - as_tags_dict (bool, default False):
If
True, return a dictionary mapping tags to pipes. Pipes with multiple tags will be repeated. - as_targets_dict (bool, default False):
If
True, return a dictionary mapping(schema, target)tuples to pipes. Pipes sharing the same target across different schemata are grouped separately. - method (str, default 'registered'):
Available options:
['registered', 'explicit', 'all']If'registered'(default), create pipes based on registered keys in the connector's pipes table (API or SQL connector, depends on mrsm_instance). If'explicit', create pipes from provided connector_keys, metric_keys, and location_keys instead of consulting the pipes table. Useful for creating non-existent pipes. If'all', create pipes from predefined metrics and locations. Requiredconnector_keys. NOTE: Method'all'is not implemented! - workers (Optional[int], default None):
If provided (and
as_tags_dictoras_targets_dictisTrue), set the number of workers for the pool to fetch tags or targets. Only takes effect if the instance connector supports multi-threading. - **kw (Any:):
Keyword arguments to pass to the
meerschaum.Pipeconstructor.
Returns
- A dictionary of dictionaries and
meerschaum.Pipeobjects - in the connector, metric, location hierarchy.
- If
as_listisTrue, return a list ofmeerschaum.Pipeobjects. - If
as_tags_dictisTrue, return a dictionary mapping tags to pipes. - If
as_targets_dictisTrue, return a dictionary mapping targets to pipes.
Examples
>>> ### Manual definition:
>>> pipes = {
... <connector_keys>: {
... <metric_key>: {
... <location_key>: Pipe(
... <connector_keys>,
... <metric_key>,
... <location_key>,
... ),
... },
... },
... },
>>> ### Accessing a single pipe:
>>> pipes['sql:main']['weather'][None]
>>> ### Return a list instead:
>>> get_pipes(as_list=True)
[Pipe('sql:main', 'weather')]
>>> get_pipes(as_tags_dict=True)
{'gvl': Pipe('sql:main', 'weather')}
68def get_connector( 69 type: str = None, 70 label: str = None, 71 refresh: bool = False, 72 debug: bool = False, 73 _load_plugins: bool = True, 74 **kw: Any 75) -> Connector: 76 """ 77 Return existing connector or create new connection and store for reuse. 78 79 You can create new connectors if enough parameters are provided for the given type and flavor. 80 81 Parameters 82 ---------- 83 type: Optional[str], default None 84 Connector type (sql, api, etc.). 85 Defaults to the type of the configured `instance_connector`. 86 87 label: Optional[str], default None 88 Connector label (e.g. main). Defaults to `'main'`. 89 90 refresh: bool, default False 91 Refresh the Connector instance / construct new object. Defaults to `False`. 92 93 kw: Any 94 Other arguments to pass to the Connector constructor. 95 If the Connector has already been constructed and new arguments are provided, 96 `refresh` is set to `True` and the old Connector is replaced. 97 98 Returns 99 ------- 100 A new Meerschaum connector (e.g. `meerschaum.connectors.api.APIConnector`, 101 `meerschaum.connectors.sql.SQLConnector`). 102 103 Examples 104 -------- 105 The following parameters would create a new 106 `meerschaum.connectors.sql.SQLConnector` that isn't in the configuration file. 107 108 ``` 109 >>> conn = get_connector( 110 ... type = 'sql', 111 ... label = 'newlabel', 112 ... flavor = 'sqlite', 113 ... database = '/file/path/to/database.db' 114 ... ) 115 >>> 116 ``` 117 118 """ 119 from meerschaum.connectors.parse import parse_instance_keys 120 from meerschaum.config import get_config 121 from meerschaum._internal.static import STATIC_CONFIG 122 from meerschaum.utils.warnings import warn 123 global _loaded_plugin_connectors 124 if isinstance(type, str) and not label and ':' in type: 125 type, label = type.split(':', maxsplit=1) 126 127 if _load_plugins: 128 with _locks['_loaded_plugin_connectors']: 129 if not _loaded_plugin_connectors: 130 load_plugin_connectors() 131 _load_builtin_custom_connectors() 132 _loaded_plugin_connectors = True 133 134 if type is None and label is None: 135 default_instance_keys = get_config('meerschaum', 'instance', patch=True) 136 ### recursive call to get_connector 137 return parse_instance_keys(default_instance_keys) 138 139 ### NOTE: the default instance connector may not be main. 140 ### Only fall back to 'main' if the type is provided by the label is omitted. 141 label = label if label is not None else STATIC_CONFIG['connectors']['default_label'] 142 143 ### type might actually be a label. Check if so and raise a warning. 144 if type not in connectors: 145 possibilities, poss_msg = [], "" 146 for _type in get_config('meerschaum', 'connectors'): 147 if type in get_config('meerschaum', 'connectors', _type): 148 possibilities.append(f"{_type}:{type}") 149 if len(possibilities) > 0: 150 poss_msg = " Did you mean" 151 for poss in possibilities[:-1]: 152 poss_msg += f" '{poss}'," 153 if poss_msg.endswith(','): 154 poss_msg = poss_msg[:-1] 155 if len(possibilities) > 1: 156 poss_msg += " or" 157 poss_msg += f" '{possibilities[-1]}'?" 158 159 warn(f"Cannot create Connector of type '{type}'." + poss_msg, stack=False) 160 return None 161 162 if 'sql' not in types: 163 from meerschaum.connectors.plugin import PluginConnector 164 from meerschaum.connectors.valkey import ValkeyConnector 165 with _locks['types']: 166 types.update({ 167 'api': APIConnector, 168 'sql': SQLConnector, 169 'plugin': PluginConnector, 170 'valkey': ValkeyConnector, 171 }) 172 173 ### determine if we need to call the constructor 174 if not refresh: 175 ### see if any user-supplied arguments differ from the existing instance 176 if label in connectors[type]: 177 warning_message = None 178 for attribute, value in kw.items(): 179 if attribute not in connectors[type][label].meta: 180 import inspect 181 cls = connectors[type][label].__class__ 182 cls_init_signature = inspect.signature(cls) 183 cls_init_params = cls_init_signature.parameters 184 if attribute not in cls_init_params: 185 warning_message = ( 186 f"Received new attribute '{attribute}' not present in connector " + 187 f"{connectors[type][label]}.\n" 188 ) 189 elif connectors[type][label].__dict__[attribute] != value: 190 warning_message = ( 191 f"Mismatched values for attribute '{attribute}' in connector " 192 + f"'{connectors[type][label]}'.\n" + 193 f" - Keyword value: '{value}'\n" + 194 f" - Existing value: '{connectors[type][label].__dict__[attribute]}'\n" 195 ) 196 if warning_message is not None: 197 warning_message += ( 198 "\nSetting `refresh` to True and recreating connector with type:" 199 + f" '{type}' and label '{label}'." 200 ) 201 refresh = True 202 warn(warning_message) 203 else: ### connector doesn't yet exist 204 refresh = True 205 206 ### only create an object if refresh is True 207 ### (can be manually specified, otherwise determined above) 208 if refresh: 209 with _locks['connectors']: 210 try: 211 ### will raise an error if configuration is incorrect / missing 212 conn = types[type](label=label, **kw) 213 connectors[type][label] = conn 214 except InvalidAttributesError as ie: 215 warn( 216 f"Incorrect attributes for connector '{type}:{label}'.\n" 217 + str(ie), 218 stack = False, 219 ) 220 conn = None 221 except Exception as e: 222 from meerschaum.utils.formatting import get_console 223 console = get_console() 224 if console: 225 console.print_exception() 226 warn( 227 f"Exception when creating connector '{type}:{label}'.\n" + str(e), 228 stack = False, 229 ) 230 conn = None 231 if conn is None: 232 return None 233 234 return connectors[type][label]
Return existing connector or create new connection and store for reuse.
You can create new connectors if enough parameters are provided for the given type and flavor.
Parameters
- type (Optional[str], default None):
Connector type (sql, api, etc.).
Defaults to the type of the configured
instance_connector. - label (Optional[str], default None):
Connector label (e.g. main). Defaults to
'main'. - refresh (bool, default False):
Refresh the Connector instance / construct new object. Defaults to
False. - kw (Any):
Other arguments to pass to the Connector constructor.
If the Connector has already been constructed and new arguments are provided,
refreshis set toTrueand the old Connector is replaced.
Returns
- A new Meerschaum connector (e.g.
meerschaum.connectors.api.APIConnector, meerschaum.connectors.sql.SQLConnector).
Examples
The following parameters would create a new
meerschaum.connectors.sql.SQLConnector that isn't in the configuration file.
>>> conn = get_connector(
... type = 'sql',
... label = 'newlabel',
... flavor = 'sqlite',
... database = '/file/path/to/database.db'
... )
>>>
112def get_config( 113 *keys: str, 114 patch: bool = True, 115 substitute: bool = True, 116 sync_files: bool = True, 117 write_missing: bool = True, 118 as_tuple: bool = False, 119 warn: bool = True, 120 debug: bool = False 121) -> Any: 122 """ 123 Return the Meerschaum configuration dictionary. 124 If positional arguments are provided, index by the keys. 125 Raises a warning if invalid keys are provided. 126 127 Parameters 128 ---------- 129 keys: str: 130 List of strings to index. 131 132 patch: bool, default True 133 If `True`, patch missing default keys into the config directory. 134 Defaults to `True`. 135 136 sync_files: bool, default True 137 If `True`, sync files if needed. 138 Defaults to `True`. 139 140 write_missing: bool, default True 141 If `True`, write default values when the main config files are missing. 142 Defaults to `True`. 143 144 substitute: bool, default True 145 If `True`, subsitute 'MRSM{}' values. 146 Defaults to `True`. 147 148 as_tuple: bool, default False 149 If `True`, return a tuple of type (success, value). 150 Defaults to `False`. 151 152 Returns 153 ------- 154 The value in the configuration directory, indexed by the provided keys. 155 156 Examples 157 -------- 158 >>> get_config('meerschaum', 'instance') 159 'sql:main' 160 >>> get_config('does', 'not', 'exist') 161 UserWarning: Invalid keys in config: ('does', 'not', 'exist') 162 """ 163 import json 164 165 symlinks_key = STATIC_CONFIG['config']['symlinks_key'] 166 if debug: 167 from meerschaum.utils.debug import dprint 168 dprint(f"Indexing keys: {keys}", color=False) 169 170 if len(keys) == 0: 171 _rc = _config( 172 substitute=substitute, 173 sync_files=sync_files, 174 write_missing=(write_missing and _allow_write_missing), 175 ) 176 if as_tuple: 177 return True, _rc 178 return _rc 179 180 ### Weird threading issues, only import if substitute is True. 181 if substitute: 182 from meerschaum.config._read_config import search_and_substitute_config 183 ### Invalidate the cache if it was read before with substitute=False 184 ### but there still exist substitutions. 185 if ( 186 config is not None and substitute and keys[0] != symlinks_key 187 and 'MRSM{' in json.dumps(config.get(keys[0])) 188 ): 189 try: 190 _subbed = search_and_substitute_config({keys[0]: config[keys[0]]}) 191 except Exception: 192 import traceback 193 traceback.print_exc() 194 _subbed = {keys[0]: config[keys[0]]} 195 196 config[keys[0]] = _subbed[keys[0]] 197 if symlinks_key in _subbed: 198 if symlinks_key not in config: 199 config[symlinks_key] = {} 200 config[symlinks_key] = apply_patch_to_config( 201 _subbed.get(symlinks_key, {}), 202 config.get(symlinks_key, {}), 203 ) 204 205 from meerschaum.config._sync import sync_files as _sync_files 206 if config is None: 207 _config(*keys, sync_files=sync_files) 208 209 invalid_keys = False 210 if keys[0] not in config and keys[0] != symlinks_key: 211 single_key_config = read_config( 212 keys=[keys[0]], substitute=substitute, write_missing=write_missing 213 ) 214 if keys[0] not in single_key_config: 215 invalid_keys = True 216 else: 217 config[keys[0]] = single_key_config.get(keys[0], None) 218 if symlinks_key in single_key_config and keys[0] in single_key_config[symlinks_key]: 219 if symlinks_key not in config: 220 config[symlinks_key] = {} 221 config[symlinks_key][keys[0]] = single_key_config[symlinks_key][keys[0]] 222 223 if sync_files: 224 _sync_files(keys=[keys[0]]) 225 226 c = config 227 if len(keys) > 0: 228 for k in keys: 229 try: 230 c = c[k] 231 except Exception: 232 invalid_keys = True 233 break 234 if invalid_keys: 235 ### Check if the keys are in the default configuration. 236 from meerschaum.config._default import default_config 237 in_default = True 238 patched_default_config = ( 239 search_and_substitute_config(default_config) 240 if substitute else copy.deepcopy(default_config) 241 ) 242 _c = patched_default_config 243 for k in keys: 244 try: 245 _c = _c[k] 246 except Exception: 247 in_default = False 248 if in_default: 249 c = _c 250 invalid_keys = False 251 warning_msg = f"Invalid keys in config: {keys}" 252 if not in_default: 253 try: 254 if warn: 255 from meerschaum.utils.warnings import warn as _warn 256 _warn(warning_msg, stacklevel=3, color=False) 257 except Exception: 258 if warn: 259 print(warning_msg) 260 if as_tuple: 261 return False, None 262 return None 263 264 ### Don't write keys that we haven't yet loaded into memory. 265 not_loaded_keys = [k for k in patched_default_config if k not in config] 266 for k in not_loaded_keys: 267 patched_default_config.pop(k, None) 268 269 set_config( 270 apply_patch_to_config( 271 patched_default_config, 272 config, 273 ) 274 ) 275 if patch and keys[0] != symlinks_key and write_missing: 276 ### Only persist defaults when the key's file is genuinely absent. 277 ### Never overwrite an existing file (e.g. one that failed to parse) ─ 278 ### doing so would clobber the user's config with default values. 279 ### Brand-new config files are still created by `read_config()`. 280 from meerschaum.config._read_config import get_keyfile_path 281 keyfile_exists = get_keyfile_path(keys[0], create_new=False) is not None 282 if not keyfile_exists: 283 write_config(config, debug=debug) 284 285 if as_tuple: 286 return (not invalid_keys), c 287 return c
Return the Meerschaum configuration dictionary. If positional arguments are provided, index by the keys. Raises a warning if invalid keys are provided.
Parameters
- keys (str:): List of strings to index.
- patch (bool, default True):
If
True, patch missing default keys into the config directory. Defaults toTrue. - sync_files (bool, default True):
If
True, sync files if needed. Defaults toTrue. - write_missing (bool, default True):
If
True, write default values when the main config files are missing. Defaults toTrue. - substitute (bool, default True):
If
True, subsitute 'MRSM{}' values. Defaults toTrue. - as_tuple (bool, default False):
If
True, return a tuple of type (success, value). Defaults toFalse.
Returns
- The value in the configuration directory, indexed by the provided keys.
Examples
>>> get_config('meerschaum', 'instance')
'sql:main'
>>> get_config('does', 'not', 'exist')
UserWarning: Invalid keys in config: ('does', 'not', 'exist')
66class Pipe: 67 """ 68 Access Meerschaum pipes via Pipe objects. 69 70 Pipes are identified by the following: 71 72 1. Connector keys (e.g. `'sql:main'`) 73 2. Metric key (e.g. `'weather'`) 74 3. Location (optional; e.g. `None`) 75 76 A pipe's connector keys correspond to a data source, and when the pipe is synced, 77 its `fetch` definition is evaluated and executed to produce new data. 78 79 Alternatively, new data may be directly synced via `pipe.sync()`: 80 81 ``` 82 >>> from meerschaum import Pipe 83 >>> pipe = Pipe('csv', 'weather') 84 >>> 85 >>> import pandas as pd 86 >>> df = pd.read_csv('weather.csv') 87 >>> pipe.sync(df) 88 ``` 89 """ 90 91 from ._fetch import ( 92 fetch, 93 get_backtrack_interval, 94 ) 95 from ._data import ( 96 get_data, 97 get_backtrack_data, 98 get_rowcount, 99 get_size, 100 get_data, 101 get_doc, 102 get_docs, 103 get_value, 104 _get_data_as_iterator, 105 get_chunk_interval, 106 _scale_delta_to_int_axis, 107 get_chunk_bounds, 108 get_chunk_bounds_batches, 109 parse_date_bounds, 110 ) 111 from ._register import register 112 from ._attributes import ( 113 attributes, 114 parameters, 115 columns, 116 indices, 117 indexes, 118 dtypes, 119 autoincrement, 120 autotime, 121 upsert, 122 static, 123 tzinfo, 124 enforce, 125 null_indices, 126 mixed_numerics, 127 get_columns, 128 get_columns_types, 129 get_columns_indices, 130 get_indices, 131 get_parameters, 132 get_dtypes, 133 update_parameters, 134 validate_indices, 135 tags, 136 get_id, 137 id, 138 get_val_column, 139 parents, 140 parent, 141 children, 142 child, 143 reference, 144 references, 145 target, 146 _target_legacy, 147 guess_datetime, 148 precision, 149 get_precision, 150 ) 151 from ._cache import ( 152 _get_cache_connector, 153 _cache_value, 154 _get_cached_value, 155 _invalidate_cache, 156 _get_cache_dir_path, 157 _write_cache_key, 158 _write_cache_file, 159 _write_cache_conn_key, 160 _read_cache_key, 161 _read_cache_file, 162 _read_cache_conn_key, 163 _load_cache_keys, 164 _load_cache_files, 165 _load_cache_conn_keys, 166 _get_cache_keys, 167 _get_cache_file_keys, 168 _get_cache_conn_keys, 169 _clear_cache_key, 170 _clear_cache_file, 171 _clear_cache_conn_key, 172 ) 173 from ._show import show 174 from ._edit import edit, edit_definition, update 175 from ._sync import ( 176 sync, 177 get_sync_time, 178 exists, 179 filter_existing, 180 _filter_existing_polars, 181 _get_chunk_label, 182 get_num_workers, 183 _persist_new_special_columns, 184 ) 185 from ._verify import ( 186 verify, 187 get_bound_interval, 188 get_bound_time, 189 ) 190 from ._delete import delete 191 from ._drop import drop, drop_indices 192 from ._compress import compress, decompress 193 from ._maintenance import vacuum, analyze, repartition 194 from ._index import create_indices 195 from ._clear import clear 196 from ._deduplicate import deduplicate 197 from ._bootstrap import bootstrap 198 from ._dtypes import enforce_dtypes, infer_dtypes 199 from ._copy import copy_to 200 201 def __init__( 202 self, 203 connector: str = '', 204 metric: str = '', 205 location: Optional[str] = None, 206 parameters: Optional[Dict[str, Any]] = None, 207 columns: Union[Dict[str, str], List[str], None] = None, 208 indices: Optional[Dict[str, Union[str, List[str]]]] = None, 209 tags: Optional[List[str]] = None, 210 target: Optional[str] = None, 211 dtypes: Optional[Dict[str, str]] = None, 212 instance: Optional[Union[str, InstanceConnector]] = None, 213 upsert: Optional[bool] = None, 214 autoincrement: Optional[bool] = None, 215 autotime: Optional[bool] = None, 216 precision: Union[str, Dict[str, Union[str, int]], None] = None, 217 static: Optional[bool] = None, 218 enforce: Optional[bool] = None, 219 null_indices: Optional[bool] = None, 220 mixed_numerics: Optional[bool] = None, 221 compress: Union[bool, Dict[str, Any], None] = None, 222 temporary: bool = False, 223 cache: Optional[bool] = None, 224 cache_connector_keys: Optional[str] = None, 225 references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None, 226 reference: Union[str, Dict[str, Any], mrsm.Pipe, None] = None, 227 parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None, 228 parent: Union[str, Dict[str, Any], mrsm.Pipe, None] = None, 229 children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None, 230 child: Union[str, Dict[str, Any], mrsm.Pipe, None] = None, 231 mrsm_instance: Optional[Union[str, InstanceConnector]] = None, 232 connector_keys: Optional[str] = None, 233 metric_key: Optional[str] = None, 234 location_key: Optional[str] = None, 235 instance_keys: Optional[str] = None, 236 indexes: Union[Dict[str, str], List[str], None] = None, 237 debug: bool = False, 238 ): 239 """ 240 Parameters 241 ---------- 242 connector: str 243 Keys for the pipe's source connector, e.g. `'sql:main'`. 244 245 metric: str 246 Label for the pipe's contents, e.g. `'weather'`. 247 248 location: str, default None 249 Label for the pipe's location. Defaults to `None`. 250 251 parameters: Optional[Dict[str, Any]], default None 252 Optionally set a pipe's parameters from the constructor, 253 e.g. columns and other attributes. 254 You can edit these parameters with `edit pipes`. 255 256 columns: Union[Dict[str, str], List[str], None], default None 257 Set the `columns` dictionary of `parameters`. 258 If `parameters` is also provided, this dictionary is added under the `'columns'` key. 259 260 indices: Optional[Dict[str, Union[str, List[str]]]], default None 261 Set the `indices` dictionary of `parameters`. 262 If `parameters` is also provided, this dictionary is added under the `'indices'` key. 263 264 tags: Optional[List[str]], default None 265 A list of strings to be added under the `'tags'` key of `parameters`. 266 You can select pipes with certain tags using `--tags`. 267 268 dtypes: Optional[Dict[str, str]], default None 269 Set the `dtypes` dictionary of `parameters`. 270 If `parameters` is also provided, this dictionary is added under the `'dtypes'` key. 271 272 mrsm_instance: Optional[Union[str, InstanceConnector]], default None 273 Connector for the Meerschaum instance where the pipe resides. 274 Defaults to the preconfigured default instance (`'sql:main'`). 275 276 instance: Optional[Union[str, InstanceConnector]], default None 277 Alias for `mrsm_instance`. If `mrsm_instance` is supplied, this value is ignored. 278 279 upsert: Optional[bool], default None 280 If `True`, set `upsert` to `True` in the parameters. 281 282 autoincrement: Optional[bool], default None 283 If `True`, set `autoincrement` in the parameters. 284 285 autotime: Optional[bool], default None 286 If `True`, set `autotime` in the parameters. 287 288 precision: Union[str, Dict[str, Union[str, int]], None], default None 289 If provided, set `precision` in the parameters. 290 This may be either a string (the precision unit) or a dictionary of in the form 291 `{'unit': <unit>, 'interval': <interval>}`. 292 Default is determined by the `datetime` column dtype 293 (e.g. `datetime64[us]` is `microsecond` precision). 294 295 static: Optional[bool], default None 296 If `True`, set `static` in the parameters. 297 298 enforce: Optional[bool], default None 299 If `False`, skip data type enforcement. 300 Default behavior is `True`. 301 302 null_indices: Optional[bool], default None 303 Set to `False` if there will be no null values in the index columns. 304 Defaults to `True`. 305 306 mixed_numerics: bool, default None 307 If `True`, integer columns will be converted to `numeric` when floats are synced. 308 Set to `False` to disable this behavior. 309 Defaults to `True`. 310 311 compress: Union[bool, Dict[str, Any], None], default None 312 If `True` (or a dictionary of compression settings), mark the pipe for compression. 313 For TimescaleDB hypertables, a columnstore (compression) policy is installed 314 automatically on sync. A dictionary may override `segmentby`, `orderby`, and `after`. 315 Defaults to `False`. 316 317 hypercore: bool, default True 318 For TimescaleDB hypertables, enable the Hypercore columnstore at table creation 319 (declaring `segmentby`/`orderby` in `CREATE TABLE`), which causes TimescaleDB to 320 auto-create a columnstore policy. Set to `False` for a plain row-store hypertable. 321 Has no effect unless the pipe is a hypertable (`hypertable`, default `True`). 322 323 temporary: bool, default False 324 If `True`, prevent instance tables (pipes, users, plugins) from being created. 325 326 cache: Optional[bool], default None 327 If `True`, cache the pipe's metadata to disk (in addition to in-memory caching). 328 If `cache` is not explicitly `True`, it is set to `False` if `temporary` is `True`. 329 Defaults to `True` (from `None`). 330 331 cache_connector_keys: Optional[str], default None 332 If provided, use the keys to a Valkey connector (e.g. `valkey:main`). 333 334 references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None 335 If provided, inherit the parameters of the reference Pipe(s). 336 May be equal to a string of the Pipe constructor, a dictionary of constructor keys, 337 a Pipe itself, or a list of any of these values. 338 339 parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None 340 Set references for parent pipes. See `references` for values. 341 342 children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None 343 Set references for child pipes. See `references` for values. 344 345 """ 346 from meerschaum.utils.warnings import error, warn 347 if (not connector and not connector_keys) or (not metric and not metric_key): 348 error( 349 "Please provide strings for the connector and metric\n " 350 + "(first two positional arguments)." 351 ) 352 353 ### Fall back to legacy `location_key` just in case. 354 if not location: 355 location = location_key 356 357 if not connector: 358 connector = connector_keys 359 360 if not metric: 361 metric = metric_key 362 363 if location in ('[None]', 'None'): 364 location = None 365 366 from meerschaum._internal.static import STATIC_CONFIG 367 negation_prefix = STATIC_CONFIG['system']['fetch_pipes_keys']['negation_prefix'] 368 for k in (connector, metric, location, *(tags or [])): 369 if str(k).startswith(negation_prefix): 370 error(f"A pipe's keys and tags cannot start with the prefix '{negation_prefix}'.") 371 372 self._connector_keys = str(connector) 373 self._connector_key = self.connector_keys ### Alias 374 self._metric_key = metric 375 self._location_key = location 376 self.temporary = temporary 377 self.cache = ( 378 cache 379 if cache is not None 380 else ((not temporary) and get_config('pipes', 'cache', 'enabled', warn=False)) 381 ) 382 self.cache_connector_keys = ( 383 str(cache_connector_keys) 384 if cache_connector_keys is not None 385 else None 386 ) 387 self.debug = debug 388 389 self._attributes: Dict[str, Any] = { 390 'connector_keys': self._connector_keys, 391 'metric_key': self._metric_key, 392 'location_key': self._location_key, 393 'parameters': {}, 394 } 395 396 ### only set parameters if values are provided 397 if isinstance(parameters, dict): 398 self._attributes['parameters'] = parameters 399 else: 400 if parameters is not None: 401 warn(f"The provided parameters are of invalid type '{type(parameters)}'.") 402 self._attributes['parameters'] = {} 403 404 columns = columns or self._attributes.get('parameters', {}).get('columns', None) 405 if isinstance(columns, (list, tuple)): 406 columns = {str(col): str(col) for col in columns} 407 if isinstance(columns, dict): 408 self._attributes['parameters']['columns'] = columns 409 elif isinstance(columns, str) and 'Pipe(' in columns: 410 pass 411 elif columns is not None: 412 warn(f"The provided columns are of invalid type '{type(columns)}'.") 413 414 indices = ( 415 indices 416 or indexes 417 or self._attributes.get('parameters', {}).get('indices', None) 418 or self._attributes.get('parameters', {}).get('indexes', None) 419 ) 420 if isinstance(indices, dict): 421 indices_key = ( 422 'indexes' 423 if 'indexes' in self._attributes['parameters'] 424 else 'indices' 425 ) 426 self._attributes['parameters'][indices_key] = indices 427 428 if isinstance(tags, (list, tuple)): 429 self._attributes['parameters']['tags'] = tags 430 elif tags is not None: 431 warn(f"The provided tags are of invalid type '{type(tags)}'.") 432 433 if isinstance(target, str): 434 self._attributes['parameters']['target'] = target 435 elif target is not None: 436 warn(f"The provided target is of invalid type '{type(target)}'.") 437 438 if isinstance(dtypes, dict): 439 self._attributes['parameters']['dtypes'] = dtypes 440 elif dtypes is not None: 441 warn(f"The provided dtypes are of invalid type '{type(dtypes)}'.") 442 443 if isinstance(upsert, bool): 444 self._attributes['parameters']['upsert'] = upsert 445 446 if isinstance(autoincrement, bool): 447 self._attributes['parameters']['autoincrement'] = autoincrement 448 449 if isinstance(autotime, bool): 450 self._attributes['parameters']['autotime'] = autotime 451 452 if isinstance(precision, dict): 453 self._attributes['parameters']['precision'] = precision 454 elif isinstance(precision, str): 455 self._attributes['parameters']['precision'] = {'unit': precision} 456 457 if isinstance(static, bool): 458 self._attributes['parameters']['static'] = static 459 self._static = static 460 461 if isinstance(enforce, bool): 462 self._attributes['parameters']['enforce'] = enforce 463 464 if isinstance(null_indices, bool): 465 self._attributes['parameters']['null_indices'] = null_indices 466 467 if isinstance(mixed_numerics, bool): 468 self._attributes['parameters']['mixed_numerics'] = mixed_numerics 469 470 if isinstance(compress, (bool, dict)): 471 self._attributes['parameters']['compress'] = compress 472 473 ### NOTE: The parameters dictionary is {} by default. 474 ### A Pipe may be registered without parameters, then edited, 475 ### or a Pipe may be registered with parameters set in-memory first. 476 _mrsm_instance = mrsm_instance if mrsm_instance is not None else (instance or instance_keys) 477 if _mrsm_instance is None: 478 _mrsm_instance = get_config('meerschaum', 'instance', patch=True) 479 480 if not isinstance(_mrsm_instance, str): 481 self._instance_connector = _mrsm_instance 482 self._instance_keys = str(_mrsm_instance) 483 else: 484 self._instance_keys = _mrsm_instance 485 486 if self._instance_keys == 'sql:memory': 487 self.cache = False 488 489 self._cache_locks = collections.defaultdict(lambda: threading.RLock()) 490 self._attributes_lock = threading.RLock() 491 492 if references is not None or reference is not None: 493 reference_vals = references if references is not None else reference 494 self.references = reference_vals 495 496 if parents is not None or parent is not None: 497 parent_vals = parents if parents is not None else parent 498 self.parents = parent_vals 499 500 if children is not None or child is not None: 501 children_vals = children if children is not None else child 502 self.children = children_vals 503 504 @property 505 def metric_key(self) -> str: 506 """ 507 Return the pipe's metric key. 508 """ 509 return self._metric_key 510 511 @property 512 def metric(self) -> str: 513 """ 514 Return the pipe's metric key. 515 """ 516 return self._metric_key 517 518 @property 519 def location_key(self) -> Union[str, None]: 520 """ 521 Return the pipe's location key. 522 """ 523 return self._location_key 524 525 @property 526 def location(self) -> Union[str, None]: 527 """ 528 Return the pipe's location key. 529 """ 530 return self._location_key 531 532 @property 533 def meta(self): 534 """ 535 Return the four keys needed to reconstruct this pipe. 536 """ 537 return { 538 'connector_keys': self.connector_keys, 539 'metric_key': self.metric_key, 540 'location_key': self.location_key, 541 'instance_keys': self.instance_keys, 542 } 543 544 def keys(self) -> List[str]: 545 """ 546 Return the ordered keys for this pipe. 547 """ 548 return { 549 key: val 550 for key, val in self.meta.items() 551 if key != 'instance' 552 } 553 554 @property 555 def instance_keys(self) -> str: 556 """ 557 Return the pipe's instance keys. 558 """ 559 return self._instance_keys 560 561 @property 562 def instance(self) -> Union[InstanceConnector, str]: 563 """ 564 Return the pipe's instance connector or keys. 565 """ 566 conn = self.instance_connector 567 if conn is None: 568 return self.instance_keys 569 return conn 570 571 @property 572 def instance_connector(self) -> Union[InstanceConnector, None]: 573 """ 574 The instance connector on which this pipe resides. 575 """ 576 if '_instance_connector' not in self.__dict__: 577 from meerschaum.connectors.parse import parse_instance_keys 578 conn = parse_instance_keys(self.instance_keys) 579 if conn: 580 self._instance_connector = conn 581 else: 582 return None 583 return self._instance_connector 584 585 @property 586 def connector_keys(self) -> str: 587 """ 588 Return the pipe's connector keys. 589 """ 590 return self._connector_keys 591 592 @property 593 def connector_key(self) -> str: 594 """ 595 Legacy: use `Pipe.connector_keys` instead. 596 """ 597 return self.connector_keys 598 599 @property 600 def connector(self) -> Union['Connector', str]: 601 """ 602 The connector to the data source. 603 """ 604 if '_connector' not in self.__dict__: 605 from meerschaum.connectors.parse import parse_instance_keys 606 import warnings 607 with warnings.catch_warnings(): 608 warnings.simplefilter('ignore') 609 try: 610 conn = parse_instance_keys(self.connector_keys) 611 except Exception: 612 conn = None 613 if conn: 614 self._connector = conn 615 else: 616 return self._connector_keys 617 return self._connector 618 619 def __str__(self, ansi: bool=False): 620 return pipe_repr(self, ansi=ansi) 621 622 def __eq__(self, other): 623 try: 624 return ( 625 isinstance(self, type(other)) 626 and self.connector_keys == other.connector_keys 627 and self.metric_key == other.metric_key 628 and self.location_key == other.location_key 629 and self.instance_keys == other.instance_keys 630 ) 631 except Exception: 632 return False 633 634 def __hash__(self): 635 ### Using an esoteric separator to avoid collisions. 636 sep = "[\"']" 637 return hash( 638 str(self.connector_keys) + sep 639 + str(self.metric_key) + sep 640 + str(self.location_key) + sep 641 + str(self.instance_keys) + sep 642 ) 643 644 def __repr__(self, ansi: bool=True, **kw) -> str: 645 if not hasattr(sys, 'ps1'): 646 ansi = False 647 648 return pipe_repr(self, ansi=ansi, **kw) 649 650 def __pt_repr__(self): 651 from meerschaum.utils.packages import attempt_import 652 prompt_toolkit_formatted_text = attempt_import('prompt_toolkit.formatted_text', lazy=False) 653 return prompt_toolkit_formatted_text.ANSI(pipe_repr(self, ansi=True)) 654 655 def __getstate__(self) -> Dict[str, Any]: 656 """ 657 Define the state dictionary (pickling). 658 """ 659 return { 660 'connector_keys': self.connector_keys, 661 'metric_key': self.metric_key, 662 'location_key': self.location_key, 663 'parameters': self._attributes.get('parameters', None), 664 'instance_keys': self.instance_keys, 665 } 666 667 def __setstate__(self, _state: Dict[str, Any]): 668 """ 669 Read the state (unpickling). 670 """ 671 self.__init__(**_state) 672 673 def __getitem__(self, key: str) -> Any: 674 """ 675 Index the pipe's attributes. 676 If the `key` cannot be found`, return `None`. 677 """ 678 if key in self.attributes: 679 return self.attributes.get(key, None) 680 681 aliases = { 682 'connector': 'connector_keys', 683 'connector_key': 'connector_keys', 684 'metric': 'metric_key', 685 'location': 'location_key', 686 } 687 aliased_key = aliases.get(key, None) 688 if aliased_key is not None: 689 return self.attributes.get(aliased_key, None) 690 691 property_aliases = { 692 'instance': 'instance_keys', 693 'instance_key': 'instance_keys', 694 } 695 aliased_key = property_aliases.get(key, None) 696 if aliased_key is not None: 697 key = aliased_key 698 return getattr(self, key, None) 699 700 def __copy__(self): 701 """ 702 Return a shallow copy of the current pipe. 703 """ 704 return mrsm.Pipe( 705 self.connector_keys, self.metric_key, self.location_key, 706 instance=self.instance_keys, 707 parameters=self._attributes.get('parameters', None), 708 ) 709 710 def __deepcopy__(self, memo): 711 """ 712 Return a deep copy of the current pipe. 713 """ 714 return self.__copy__()
Access Meerschaum pipes via Pipe objects.
Pipes are identified by the following:
- Connector keys (e.g.
'sql:main') - Metric key (e.g.
'weather') - Location (optional; e.g.
None)
A pipe's connector keys correspond to a data source, and when the pipe is synced,
its fetch definition is evaluated and executed to produce new data.
Alternatively, new data may be directly synced via pipe.sync():
>>> from meerschaum import Pipe
>>> pipe = Pipe('csv', 'weather')
>>>
>>> import pandas as pd
>>> df = pd.read_csv('weather.csv')
>>> pipe.sync(df)
201 def __init__( 202 self, 203 connector: str = '', 204 metric: str = '', 205 location: Optional[str] = None, 206 parameters: Optional[Dict[str, Any]] = None, 207 columns: Union[Dict[str, str], List[str], None] = None, 208 indices: Optional[Dict[str, Union[str, List[str]]]] = None, 209 tags: Optional[List[str]] = None, 210 target: Optional[str] = None, 211 dtypes: Optional[Dict[str, str]] = None, 212 instance: Optional[Union[str, InstanceConnector]] = None, 213 upsert: Optional[bool] = None, 214 autoincrement: Optional[bool] = None, 215 autotime: Optional[bool] = None, 216 precision: Union[str, Dict[str, Union[str, int]], None] = None, 217 static: Optional[bool] = None, 218 enforce: Optional[bool] = None, 219 null_indices: Optional[bool] = None, 220 mixed_numerics: Optional[bool] = None, 221 compress: Union[bool, Dict[str, Any], None] = None, 222 temporary: bool = False, 223 cache: Optional[bool] = None, 224 cache_connector_keys: Optional[str] = None, 225 references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None, 226 reference: Union[str, Dict[str, Any], mrsm.Pipe, None] = None, 227 parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None, 228 parent: Union[str, Dict[str, Any], mrsm.Pipe, None] = None, 229 children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]] = None, 230 child: Union[str, Dict[str, Any], mrsm.Pipe, None] = None, 231 mrsm_instance: Optional[Union[str, InstanceConnector]] = None, 232 connector_keys: Optional[str] = None, 233 metric_key: Optional[str] = None, 234 location_key: Optional[str] = None, 235 instance_keys: Optional[str] = None, 236 indexes: Union[Dict[str, str], List[str], None] = None, 237 debug: bool = False, 238 ): 239 """ 240 Parameters 241 ---------- 242 connector: str 243 Keys for the pipe's source connector, e.g. `'sql:main'`. 244 245 metric: str 246 Label for the pipe's contents, e.g. `'weather'`. 247 248 location: str, default None 249 Label for the pipe's location. Defaults to `None`. 250 251 parameters: Optional[Dict[str, Any]], default None 252 Optionally set a pipe's parameters from the constructor, 253 e.g. columns and other attributes. 254 You can edit these parameters with `edit pipes`. 255 256 columns: Union[Dict[str, str], List[str], None], default None 257 Set the `columns` dictionary of `parameters`. 258 If `parameters` is also provided, this dictionary is added under the `'columns'` key. 259 260 indices: Optional[Dict[str, Union[str, List[str]]]], default None 261 Set the `indices` dictionary of `parameters`. 262 If `parameters` is also provided, this dictionary is added under the `'indices'` key. 263 264 tags: Optional[List[str]], default None 265 A list of strings to be added under the `'tags'` key of `parameters`. 266 You can select pipes with certain tags using `--tags`. 267 268 dtypes: Optional[Dict[str, str]], default None 269 Set the `dtypes` dictionary of `parameters`. 270 If `parameters` is also provided, this dictionary is added under the `'dtypes'` key. 271 272 mrsm_instance: Optional[Union[str, InstanceConnector]], default None 273 Connector for the Meerschaum instance where the pipe resides. 274 Defaults to the preconfigured default instance (`'sql:main'`). 275 276 instance: Optional[Union[str, InstanceConnector]], default None 277 Alias for `mrsm_instance`. If `mrsm_instance` is supplied, this value is ignored. 278 279 upsert: Optional[bool], default None 280 If `True`, set `upsert` to `True` in the parameters. 281 282 autoincrement: Optional[bool], default None 283 If `True`, set `autoincrement` in the parameters. 284 285 autotime: Optional[bool], default None 286 If `True`, set `autotime` in the parameters. 287 288 precision: Union[str, Dict[str, Union[str, int]], None], default None 289 If provided, set `precision` in the parameters. 290 This may be either a string (the precision unit) or a dictionary of in the form 291 `{'unit': <unit>, 'interval': <interval>}`. 292 Default is determined by the `datetime` column dtype 293 (e.g. `datetime64[us]` is `microsecond` precision). 294 295 static: Optional[bool], default None 296 If `True`, set `static` in the parameters. 297 298 enforce: Optional[bool], default None 299 If `False`, skip data type enforcement. 300 Default behavior is `True`. 301 302 null_indices: Optional[bool], default None 303 Set to `False` if there will be no null values in the index columns. 304 Defaults to `True`. 305 306 mixed_numerics: bool, default None 307 If `True`, integer columns will be converted to `numeric` when floats are synced. 308 Set to `False` to disable this behavior. 309 Defaults to `True`. 310 311 compress: Union[bool, Dict[str, Any], None], default None 312 If `True` (or a dictionary of compression settings), mark the pipe for compression. 313 For TimescaleDB hypertables, a columnstore (compression) policy is installed 314 automatically on sync. A dictionary may override `segmentby`, `orderby`, and `after`. 315 Defaults to `False`. 316 317 hypercore: bool, default True 318 For TimescaleDB hypertables, enable the Hypercore columnstore at table creation 319 (declaring `segmentby`/`orderby` in `CREATE TABLE`), which causes TimescaleDB to 320 auto-create a columnstore policy. Set to `False` for a plain row-store hypertable. 321 Has no effect unless the pipe is a hypertable (`hypertable`, default `True`). 322 323 temporary: bool, default False 324 If `True`, prevent instance tables (pipes, users, plugins) from being created. 325 326 cache: Optional[bool], default None 327 If `True`, cache the pipe's metadata to disk (in addition to in-memory caching). 328 If `cache` is not explicitly `True`, it is set to `False` if `temporary` is `True`. 329 Defaults to `True` (from `None`). 330 331 cache_connector_keys: Optional[str], default None 332 If provided, use the keys to a Valkey connector (e.g. `valkey:main`). 333 334 references: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None 335 If provided, inherit the parameters of the reference Pipe(s). 336 May be equal to a string of the Pipe constructor, a dictionary of constructor keys, 337 a Pipe itself, or a list of any of these values. 338 339 parents: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None 340 Set references for parent pipes. See `references` for values. 341 342 children: Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None 343 Set references for child pipes. See `references` for values. 344 345 """ 346 from meerschaum.utils.warnings import error, warn 347 if (not connector and not connector_keys) or (not metric and not metric_key): 348 error( 349 "Please provide strings for the connector and metric\n " 350 + "(first two positional arguments)." 351 ) 352 353 ### Fall back to legacy `location_key` just in case. 354 if not location: 355 location = location_key 356 357 if not connector: 358 connector = connector_keys 359 360 if not metric: 361 metric = metric_key 362 363 if location in ('[None]', 'None'): 364 location = None 365 366 from meerschaum._internal.static import STATIC_CONFIG 367 negation_prefix = STATIC_CONFIG['system']['fetch_pipes_keys']['negation_prefix'] 368 for k in (connector, metric, location, *(tags or [])): 369 if str(k).startswith(negation_prefix): 370 error(f"A pipe's keys and tags cannot start with the prefix '{negation_prefix}'.") 371 372 self._connector_keys = str(connector) 373 self._connector_key = self.connector_keys ### Alias 374 self._metric_key = metric 375 self._location_key = location 376 self.temporary = temporary 377 self.cache = ( 378 cache 379 if cache is not None 380 else ((not temporary) and get_config('pipes', 'cache', 'enabled', warn=False)) 381 ) 382 self.cache_connector_keys = ( 383 str(cache_connector_keys) 384 if cache_connector_keys is not None 385 else None 386 ) 387 self.debug = debug 388 389 self._attributes: Dict[str, Any] = { 390 'connector_keys': self._connector_keys, 391 'metric_key': self._metric_key, 392 'location_key': self._location_key, 393 'parameters': {}, 394 } 395 396 ### only set parameters if values are provided 397 if isinstance(parameters, dict): 398 self._attributes['parameters'] = parameters 399 else: 400 if parameters is not None: 401 warn(f"The provided parameters are of invalid type '{type(parameters)}'.") 402 self._attributes['parameters'] = {} 403 404 columns = columns or self._attributes.get('parameters', {}).get('columns', None) 405 if isinstance(columns, (list, tuple)): 406 columns = {str(col): str(col) for col in columns} 407 if isinstance(columns, dict): 408 self._attributes['parameters']['columns'] = columns 409 elif isinstance(columns, str) and 'Pipe(' in columns: 410 pass 411 elif columns is not None: 412 warn(f"The provided columns are of invalid type '{type(columns)}'.") 413 414 indices = ( 415 indices 416 or indexes 417 or self._attributes.get('parameters', {}).get('indices', None) 418 or self._attributes.get('parameters', {}).get('indexes', None) 419 ) 420 if isinstance(indices, dict): 421 indices_key = ( 422 'indexes' 423 if 'indexes' in self._attributes['parameters'] 424 else 'indices' 425 ) 426 self._attributes['parameters'][indices_key] = indices 427 428 if isinstance(tags, (list, tuple)): 429 self._attributes['parameters']['tags'] = tags 430 elif tags is not None: 431 warn(f"The provided tags are of invalid type '{type(tags)}'.") 432 433 if isinstance(target, str): 434 self._attributes['parameters']['target'] = target 435 elif target is not None: 436 warn(f"The provided target is of invalid type '{type(target)}'.") 437 438 if isinstance(dtypes, dict): 439 self._attributes['parameters']['dtypes'] = dtypes 440 elif dtypes is not None: 441 warn(f"The provided dtypes are of invalid type '{type(dtypes)}'.") 442 443 if isinstance(upsert, bool): 444 self._attributes['parameters']['upsert'] = upsert 445 446 if isinstance(autoincrement, bool): 447 self._attributes['parameters']['autoincrement'] = autoincrement 448 449 if isinstance(autotime, bool): 450 self._attributes['parameters']['autotime'] = autotime 451 452 if isinstance(precision, dict): 453 self._attributes['parameters']['precision'] = precision 454 elif isinstance(precision, str): 455 self._attributes['parameters']['precision'] = {'unit': precision} 456 457 if isinstance(static, bool): 458 self._attributes['parameters']['static'] = static 459 self._static = static 460 461 if isinstance(enforce, bool): 462 self._attributes['parameters']['enforce'] = enforce 463 464 if isinstance(null_indices, bool): 465 self._attributes['parameters']['null_indices'] = null_indices 466 467 if isinstance(mixed_numerics, bool): 468 self._attributes['parameters']['mixed_numerics'] = mixed_numerics 469 470 if isinstance(compress, (bool, dict)): 471 self._attributes['parameters']['compress'] = compress 472 473 ### NOTE: The parameters dictionary is {} by default. 474 ### A Pipe may be registered without parameters, then edited, 475 ### or a Pipe may be registered with parameters set in-memory first. 476 _mrsm_instance = mrsm_instance if mrsm_instance is not None else (instance or instance_keys) 477 if _mrsm_instance is None: 478 _mrsm_instance = get_config('meerschaum', 'instance', patch=True) 479 480 if not isinstance(_mrsm_instance, str): 481 self._instance_connector = _mrsm_instance 482 self._instance_keys = str(_mrsm_instance) 483 else: 484 self._instance_keys = _mrsm_instance 485 486 if self._instance_keys == 'sql:memory': 487 self.cache = False 488 489 self._cache_locks = collections.defaultdict(lambda: threading.RLock()) 490 self._attributes_lock = threading.RLock() 491 492 if references is not None or reference is not None: 493 reference_vals = references if references is not None else reference 494 self.references = reference_vals 495 496 if parents is not None or parent is not None: 497 parent_vals = parents if parents is not None else parent 498 self.parents = parent_vals 499 500 if children is not None or child is not None: 501 children_vals = children if children is not None else child 502 self.children = children_vals
Parameters
- connector (str):
Keys for the pipe's source connector, e.g.
'sql:main'. - metric (str):
Label for the pipe's contents, e.g.
'weather'. - location (str, default None):
Label for the pipe's location. Defaults to
None. - parameters (Optional[Dict[str, Any]], default None):
Optionally set a pipe's parameters from the constructor,
e.g. columns and other attributes.
You can edit these parameters with
edit pipes. - columns (Union[Dict[str, str], List[str], None], default None):
Set the
columnsdictionary ofparameters. Ifparametersis also provided, this dictionary is added under the'columns'key. - indices (Optional[Dict[str, Union[str, List[str]]]], default None):
Set the
indicesdictionary ofparameters. Ifparametersis also provided, this dictionary is added under the'indices'key. - tags (Optional[List[str]], default None):
A list of strings to be added under the
'tags'key ofparameters. You can select pipes with certain tags using--tags. - dtypes (Optional[Dict[str, str]], default None):
Set the
dtypesdictionary ofparameters. Ifparametersis also provided, this dictionary is added under the'dtypes'key. - mrsm_instance (Optional[Union[str, InstanceConnector]], default None):
Connector for the Meerschaum instance where the pipe resides.
Defaults to the preconfigured default instance (
'sql:main'). - instance (Optional[Union[str, InstanceConnector]], default None):
Alias for
mrsm_instance. Ifmrsm_instanceis supplied, this value is ignored. - upsert (Optional[bool], default None):
If
True, setupserttoTruein the parameters. - autoincrement (Optional[bool], default None):
If
True, setautoincrementin the parameters. - autotime (Optional[bool], default None):
If
True, setautotimein the parameters. - precision (Union[str, Dict[str, Union[str, int]], None], default None):
If provided, set
precisionin the parameters. This may be either a string (the precision unit) or a dictionary of in the form{'unit': <unit>, 'interval': <interval>}. Default is determined by thedatetimecolumn dtype (e.g.datetime64[us]ismicrosecondprecision). - static (Optional[bool], default None):
If
True, setstaticin the parameters. - enforce (Optional[bool], default None):
If
False, skip data type enforcement. Default behavior isTrue. - null_indices (Optional[bool], default None):
Set to
Falseif there will be no null values in the index columns. Defaults toTrue. - mixed_numerics (bool, default None):
If
True, integer columns will be converted tonumericwhen floats are synced. Set toFalseto disable this behavior. Defaults toTrue. - compress (Union[bool, Dict[str, Any], None], default None):
If
True(or a dictionary of compression settings), mark the pipe for compression. For TimescaleDB hypertables, a columnstore (compression) policy is installed automatically on sync. A dictionary may overridesegmentby,orderby, andafter. Defaults toFalse. - hypercore (bool, default True):
For TimescaleDB hypertables, enable the Hypercore columnstore at table creation
(declaring
segmentby/orderbyinCREATE TABLE), which causes TimescaleDB to auto-create a columnstore policy. Set toFalsefor a plain row-store hypertable. Has no effect unless the pipe is a hypertable (hypertable, defaultTrue). - temporary (bool, default False):
If
True, prevent instance tables (pipes, users, plugins) from being created. - cache (Optional[bool], default None):
If
True, cache the pipe's metadata to disk (in addition to in-memory caching). Ifcacheis not explicitlyTrue, it is set toFalseiftemporaryisTrue. Defaults toTrue(fromNone). - cache_connector_keys (Optional[str], default None):
If provided, use the keys to a Valkey connector (e.g.
valkey:main). - references (Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None): If provided, inherit the parameters of the reference Pipe(s). May be equal to a string of the Pipe constructor, a dictionary of constructor keys, a Pipe itself, or a list of any of these values.
- parents (Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None):
Set references for parent pipes. See
referencesfor values. - children (Optional[List[Union[str, Dict[str, Any], mrsm.Pipe, None]]], default None):
Set references for child pipes. See
referencesfor values.
504 @property 505 def metric_key(self) -> str: 506 """ 507 Return the pipe's metric key. 508 """ 509 return self._metric_key
Return the pipe's metric key.
511 @property 512 def metric(self) -> str: 513 """ 514 Return the pipe's metric key. 515 """ 516 return self._metric_key
Return the pipe's metric key.
518 @property 519 def location_key(self) -> Union[str, None]: 520 """ 521 Return the pipe's location key. 522 """ 523 return self._location_key
Return the pipe's location key.
525 @property 526 def location(self) -> Union[str, None]: 527 """ 528 Return the pipe's location key. 529 """ 530 return self._location_key
Return the pipe's location key.
532 @property 533 def meta(self): 534 """ 535 Return the four keys needed to reconstruct this pipe. 536 """ 537 return { 538 'connector_keys': self.connector_keys, 539 'metric_key': self.metric_key, 540 'location_key': self.location_key, 541 'instance_keys': self.instance_keys, 542 }
Return the four keys needed to reconstruct this pipe.
544 def keys(self) -> List[str]: 545 """ 546 Return the ordered keys for this pipe. 547 """ 548 return { 549 key: val 550 for key, val in self.meta.items() 551 if key != 'instance' 552 }
Return the ordered keys for this pipe.
554 @property 555 def instance_keys(self) -> str: 556 """ 557 Return the pipe's instance keys. 558 """ 559 return self._instance_keys
Return the pipe's instance keys.
561 @property 562 def instance(self) -> Union[InstanceConnector, str]: 563 """ 564 Return the pipe's instance connector or keys. 565 """ 566 conn = self.instance_connector 567 if conn is None: 568 return self.instance_keys 569 return conn
Return the pipe's instance connector or keys.
571 @property 572 def instance_connector(self) -> Union[InstanceConnector, None]: 573 """ 574 The instance connector on which this pipe resides. 575 """ 576 if '_instance_connector' not in self.__dict__: 577 from meerschaum.connectors.parse import parse_instance_keys 578 conn = parse_instance_keys(self.instance_keys) 579 if conn: 580 self._instance_connector = conn 581 else: 582 return None 583 return self._instance_connector
The instance connector on which this pipe resides.
585 @property 586 def connector_keys(self) -> str: 587 """ 588 Return the pipe's connector keys. 589 """ 590 return self._connector_keys
Return the pipe's connector keys.
592 @property 593 def connector_key(self) -> str: 594 """ 595 Legacy: use `Pipe.connector_keys` instead. 596 """ 597 return self.connector_keys
Legacy: use Pipe.connector_keys instead.
599 @property 600 def connector(self) -> Union['Connector', str]: 601 """ 602 The connector to the data source. 603 """ 604 if '_connector' not in self.__dict__: 605 from meerschaum.connectors.parse import parse_instance_keys 606 import warnings 607 with warnings.catch_warnings(): 608 warnings.simplefilter('ignore') 609 try: 610 conn = parse_instance_keys(self.connector_keys) 611 except Exception: 612 conn = None 613 if conn: 614 self._connector = conn 615 else: 616 return self._connector_keys 617 return self._connector
The connector to the data source.
21def fetch( 22 self, 23 begin: Union[datetime, int, str, None] = '', 24 end: Union[datetime, int, None] = None, 25 check_existing: bool = True, 26 sync_chunks: bool = False, 27 debug: bool = False, 28 **kw: Any 29) -> Union['pd.DataFrame', Iterator['pd.DataFrame'], None]: 30 """ 31 Fetch a Pipe's latest data from its connector. 32 33 Parameters 34 ---------- 35 begin: Union[datetime, str, None], default '': 36 If provided, only fetch data newer than or equal to `begin`. 37 38 end: Optional[datetime], default None: 39 If provided, only fetch data older than or equal to `end`. 40 41 check_existing: bool, default True 42 If `False`, do not apply the backtrack interval. 43 44 sync_chunks: bool, default False 45 If `True` and the pipe's connector is of type `'sql'`, begin syncing chunks while fetching 46 loads chunks into memory. 47 48 debug: bool, default False 49 Verbosity toggle. 50 51 Returns 52 ------- 53 A `pd.DataFrame` of the newest unseen data. 54 55 """ 56 if 'fetch' not in dir(self.connector): 57 warn(f"No `fetch()` function defined for connector '{self.connector}'") 58 return None 59 60 from meerschaum.connectors import get_connector_plugin 61 from meerschaum.utils.misc import filter_arguments 62 63 _chunk_hook = kw.pop('chunk_hook', None) 64 kw['workers'] = self.get_num_workers(kw.get('workers', None)) 65 if sync_chunks and _chunk_hook is None: 66 67 def _chunk_hook(chunk, **_kw) -> SuccessTuple: 68 """ 69 Wrap `Pipe.sync()` with a custom chunk label prepended to the message. 70 """ 71 from meerschaum.config._patch import apply_patch_to_config 72 kwargs = apply_patch_to_config(kw, _kw) 73 chunk_success, chunk_message = self.sync(chunk, **kwargs) 74 chunk_label = self._get_chunk_label(chunk, self.columns.get('datetime', None)) 75 if chunk_label: 76 chunk_message = '\n' + chunk_label + '\n' + chunk_message 77 return chunk_success, chunk_message 78 79 begin, end = self.parse_date_bounds(begin, end) 80 81 with mrsm.Venv(get_connector_plugin(self.connector)): 82 _args, _kwargs = filter_arguments( 83 self.connector.fetch, 84 self, 85 begin=_determine_begin( 86 self, 87 begin, 88 end, 89 check_existing=check_existing, 90 debug=debug, 91 ), 92 end=end, 93 chunk_hook=_chunk_hook, 94 debug=debug, 95 **kw 96 ) 97 df = self.connector.fetch(*_args, **_kwargs) 98 return df
Fetch a Pipe's latest data from its connector.
Parameters
- begin (Union[datetime, str, None], default '':):
If provided, only fetch data newer than or equal to
begin. - end (Optional[datetime], default None:):
If provided, only fetch data older than or equal to
end. - check_existing (bool, default True):
If
False, do not apply the backtrack interval. - sync_chunks (bool, default False):
If
Trueand the pipe's connector is of type'sql', begin syncing chunks while fetching loads chunks into memory. - debug (bool, default False): Verbosity toggle.
Returns
- A
pd.DataFrameof the newest unseen data.
101def get_backtrack_interval( 102 self, 103 check_existing: bool = True, 104 debug: bool = False, 105) -> Union[timedelta, int]: 106 """ 107 Get the chunk interval to use for this pipe. 108 109 Parameters 110 ---------- 111 check_existing: bool, default True 112 If `False`, return a backtrack_interval of 0 minutes. 113 114 Returns 115 ------- 116 The backtrack interval (`timedelta` or `int`) to use with this pipe's `datetime` axis. 117 """ 118 from meerschaum.utils.dtypes import MRSM_PRECISION_UNITS_SCALARS, MRSM_PRECISION_UNITS_ALIASES 119 default_backtrack_minutes = get_config('pipes', 'parameters', 'fetch', 'backtrack_minutes') 120 configured_backtrack_minutes = self.parameters.get('fetch', {}).get('backtrack_minutes', None) 121 backtrack_minutes = ( 122 configured_backtrack_minutes 123 if configured_backtrack_minutes is not None 124 else default_backtrack_minutes 125 ) if check_existing else 0 126 127 dt_col = self.columns.get('datetime', None) 128 if dt_col is None: 129 return timedelta(minutes=backtrack_minutes) 130 131 dt_dtype = self.dtypes.get(dt_col, 'datetime') 132 if 'int' in dt_dtype.lower(): 133 if not self.parameters.get('precision', None): 134 return backtrack_minutes 135 precision_unit = self.precision.get('unit', None) 136 true_unit = MRSM_PRECISION_UNITS_ALIASES.get(precision_unit, precision_unit) 137 scalar = MRSM_PRECISION_UNITS_SCALARS.get(true_unit, None) 138 if scalar is not None: 139 return int(backtrack_minutes * 60 * scalar) 140 return backtrack_minutes 141 142 return timedelta(minutes=backtrack_minutes)
Get the chunk interval to use for this pipe.
Parameters
- check_existing (bool, default True):
If
False, return a backtrack_interval of 0 minutes.
Returns
- The backtrack interval (
timedeltaorint) to use with this pipe'sdatetimeaxis.
23def get_data( 24 self, 25 select_columns: Optional[List[str]] = None, 26 omit_columns: Optional[List[str]] = None, 27 begin: Union[datetime, int, str, None] = None, 28 end: Union[datetime, int, str, None] = None, 29 params: Optional[Dict[str, Any]] = None, 30 as_docs: bool = False, 31 as_iterator: bool = False, 32 as_chunks: bool = False, 33 as_dask: bool = False, 34 as_polars: bool = False, 35 add_missing_columns: bool = False, 36 chunk_interval: Union[timedelta, int, None] = None, 37 order: Optional[str] = 'asc', 38 limit: Optional[int] = None, 39 fresh: bool = False, 40 debug: bool = False, 41 **kw: Any 42) -> Union['pd.DataFrame', Iterator['pd.DataFrame'], None]: 43 """ 44 Get a pipe's data from the instance connector. 45 46 Parameters 47 ---------- 48 select_columns: Optional[List[str]], default None 49 If provided, only select these given columns. 50 Otherwise select all available columns (i.e. `SELECT *`). 51 52 omit_columns: Optional[List[str]], default None 53 If provided, remove these columns from the selection. 54 55 begin: Union[datetime, int, str, None], default None 56 Lower bound datetime to begin searching for data (inclusive). 57 Translates to a `WHERE` clause like `WHERE datetime >= begin`. 58 Defaults to `None`. 59 60 end: Union[datetime, int, str, None], default None 61 Upper bound datetime to stop searching for data (inclusive). 62 Translates to a `WHERE` clause like `WHERE datetime < end`. 63 Defaults to `None`. 64 65 params: Optional[Dict[str, Any]], default None 66 Filter the retrieved data by a dictionary of parameters. 67 See `meerschaum.utils.sql.build_where` for more details. 68 69 as_docs: bool, default False 70 If `True`, return a list of dictionaries rather than a DataFrame. 71 Relies on `get_pipe_docs` from the instance connector if implemented. 72 May be combined with `as_chunks` to return an `Iterator[List[Dict]]` 73 chunked by time bounds (useful for large result sets without pandas overhead). 74 75 as_iterator: bool, default False 76 If `True`, return a generator of chunks of pipe data. 77 When combined with `as_docs=True`, yields `List[Dict]` per chunk instead of DataFrames. 78 79 as_chunks: bool, default False 80 Alias for `as_iterator`. 81 When combined with `as_docs=True`, yields `List[Dict]` per chunk instead of DataFrames. 82 83 as_dask: bool, default False 84 If `True`, return a `dask.DataFrame` 85 (which may be loaded into a Pandas DataFrame with `df.compute()`). 86 87 as_polars: bool, default False 88 If `True`, return a `polars.DataFrame`. When combined with `as_iterator=True`, 89 yield Polars DataFrames. This may not be combined with `as_docs` or `as_dask`. 90 91 add_missing_columns: bool, default False 92 If `True`, add any missing columns from `Pipe.dtypes` to the dataframe. 93 94 chunk_interval: Union[timedelta, int, None], default None 95 If `as_iterator`, then return chunks with `begin` and `end` separated by this interval. 96 This may be set under `pipe.parameters['chunk_minutes']`. 97 By default, use a timedelta of 43200 minutes (30 days). 98 If `chunk_interval` is an integer and the `datetime` axis a timestamp, 99 then use a timedelta with the number of minutes configured to this value. 100 If the `datetime` axis is an integer, an `int` is used verbatim as the number of 101 epoch units, and a `timedelta` is scaled into epoch units by the pipe's `precision`. 102 103 order: Optional[str], default 'asc' 104 If `order` is not `None`, sort the resulting dataframe by indices. 105 106 limit: Optional[int], default None 107 If provided, cap the dataframe to this many rows. 108 109 fresh: bool, default False 110 If `True`, skip local cache and directly query the instance connector. 111 112 debug: bool, default False 113 Verbosity toggle. 114 Defaults to `False`. 115 116 Returns 117 ------- 118 A `pd.DataFrame` of the pipe's data (default). 119 A `List[Dict]` if `as_docs=True`. 120 An `Iterator[pd.DataFrame]` if `as_chunks=True` (or `as_iterator=True`). 121 An `Iterator[List[Dict]]` if both `as_docs=True` and `as_chunks=True`. 122 A `polars.DataFrame` if `as_polars=True`. 123 124 """ 125 from meerschaum.utils.warnings import warn 126 from meerschaum.utils.venv import Venv 127 from meerschaum.connectors import get_connector_plugin 128 from meerschaum.utils.dtypes import to_pandas_dtype 129 from meerschaum.utils.dataframe import add_missing_cols_to_df, df_is_chunk_generator 130 from meerschaum.utils.packages import attempt_import 131 from meerschaum.utils.warnings import dprint 132 dd = attempt_import('dask.dataframe') if as_dask else None 133 dask = attempt_import('dask') if as_dask else None 134 _ = attempt_import('partd', lazy=False) if as_dask else None 135 136 if as_polars and (as_docs or as_dask): 137 raise ValueError("`as_polars` may not be combined with `as_docs` or `as_dask`.") 138 139 if select_columns == '*': 140 select_columns = None 141 elif isinstance(select_columns, str): 142 select_columns = [select_columns] 143 144 if isinstance(omit_columns, str): 145 omit_columns = [omit_columns] 146 147 begin, end = self.parse_date_bounds(begin, end, debug=debug) 148 as_iterator = as_iterator or as_chunks 149 dt_col = self.columns.get('datetime', None) 150 151 def _sort_df(_df): 152 if df_is_chunk_generator(_df): 153 return _df 154 indices = [] if dt_col not in _df.columns else [dt_col] 155 non_dt_cols = [ 156 col 157 for col_ix, col in self.columns.items() 158 if col_ix != 'datetime' and col in _df.columns 159 ] 160 indices.extend(non_dt_cols) 161 if _df.__module__.split('.')[0] == 'polars': 162 _df = _df.sort( 163 indices, 164 descending=(str(order).lower() != 'asc'), 165 ) 166 elif 'dask' not in _df.__module__: 167 ascending = str(order).lower() == 'asc' 168 dt_series = _df[dt_col] if indices and indices[0] == dt_col else None 169 ### Unique, non-null, ordered datetimes (e.g. from the SQL `ORDER BY`) are already sorted. 170 already_sorted = ( 171 dt_series is not None 172 and not dt_series.hasnans 173 and ( 174 dt_series.is_monotonic_increasing 175 if ascending 176 else dt_series.is_monotonic_decreasing 177 ) 178 and dt_series.is_unique 179 ) 180 if not already_sorted: 181 _df.sort_values( 182 by=indices, 183 inplace=True, 184 ascending=ascending, 185 ) 186 _df.reset_index(drop=True, inplace=True) 187 else: 188 _df = _df.sort_values( 189 by=indices, 190 ascending=(str(order).lower() == 'asc'), 191 ) 192 _df = _df.reset_index(drop=True) 193 if limit is not None and len(_df) > limit: 194 return _df.head(limit) 195 return _df 196 197 if as_iterator or as_chunks: 198 df = self._get_data_as_iterator( 199 select_columns=select_columns, 200 omit_columns=omit_columns, 201 begin=begin, 202 end=end, 203 params=params, 204 chunk_interval=chunk_interval, 205 limit=limit, 206 order=order, 207 as_docs=as_docs, 208 as_polars=as_polars, 209 fresh=fresh, 210 debug=debug, 211 ) 212 if as_docs: 213 return df 214 df = _sort_df(df) 215 return df 216 217 if as_dask: 218 from multiprocessing.pool import ThreadPool 219 dask_pool = ThreadPool(self.get_num_workers()) 220 dask.config.set(pool=dask_pool) 221 chunk_interval = self.get_chunk_interval(chunk_interval, debug=debug) 222 bounds = self.get_chunk_bounds( 223 begin=begin, 224 end=end, 225 bounded=False, 226 chunk_interval=chunk_interval, 227 debug=debug, 228 ) 229 dask_chunks = [ 230 dask.delayed(self.get_data)( 231 select_columns=select_columns, 232 omit_columns=omit_columns, 233 begin=chunk_begin, 234 end=chunk_end, 235 params=params, 236 chunk_interval=chunk_interval, 237 order=order, 238 limit=limit, 239 fresh=fresh, 240 add_missing_columns=True, 241 debug=debug, 242 ) 243 for (chunk_begin, chunk_end) in bounds 244 ] 245 dask_meta = { 246 col: to_pandas_dtype(typ) 247 for col, typ in self.get_dtypes(refresh=True, infer=True, debug=debug).items() 248 } 249 if debug: 250 dprint(f"Dask meta:\n{dask_meta}") 251 dask_df = dd.from_delayed(dask_chunks, meta=dask_meta, verify_meta=False) 252 return _sort_df(dask_df.astype(dask_meta)) 253 254 if not self.exists(debug=debug): 255 return [] if as_docs else None 256 257 if as_docs: 258 with Venv(get_connector_plugin(self.instance_connector)): 259 docs = self.instance_connector.get_pipe_docs( 260 pipe=self, 261 select_columns=select_columns, 262 omit_columns=omit_columns, 263 begin=begin, 264 end=end, 265 params=params, 266 limit=limit, 267 order=order, 268 debug=debug, 269 **kw 270 ) 271 return docs if docs is not None else [] 272 273 with Venv(get_connector_plugin(self.instance_connector)): 274 df = self.instance_connector.get_pipe_data( 275 pipe=self, 276 select_columns=select_columns, 277 omit_columns=omit_columns, 278 begin=begin, 279 end=end, 280 params=params, 281 limit=limit, 282 order=order, 283 debug=debug, 284 **( 285 {'as_polars': True} 286 if as_polars and self.instance_connector.type == 'sql' 287 else {} 288 ), 289 **kw 290 ) 291 if df is None: 292 return df 293 294 if not select_columns: 295 select_columns = [col for col in df.columns] 296 297 pipe_dtypes = self.get_dtypes(refresh=False, debug=debug) 298 cols_to_omit = [ 299 col 300 for col in df.columns 301 if ( 302 col in (omit_columns or []) 303 or 304 col not in (select_columns or []) 305 ) 306 ] 307 cols_to_add = [ 308 col 309 for col in select_columns 310 if col not in df.columns 311 ] + ([ 312 col 313 for col in pipe_dtypes 314 if col not in df.columns 315 ] if add_missing_columns else []) 316 if cols_to_omit: 317 warn( 318 ( 319 f"Received {len(cols_to_omit)} omitted column" 320 + ('s' if len(cols_to_omit) != 1 else '') 321 + f" for {self}. " 322 + "Consider adding `select_columns` and `omit_columns` support to " 323 + f"'{self.instance_connector.type}' connectors to improve performance." 324 ), 325 stack=False, 326 ) 327 _cols_to_select = [col for col in df.columns if col not in cols_to_omit] 328 df = df[_cols_to_select] 329 330 if cols_to_add: 331 if not add_missing_columns: 332 from meerschaum.utils.misc import items_str 333 warn( 334 f"Will add columns {items_str(cols_to_add)} as nulls to dataframe.", 335 stack=False, 336 ) 337 338 df = add_missing_cols_to_df( 339 df, 340 { 341 col: pipe_dtypes.get(col, 'string') 342 for col in cols_to_add 343 }, 344 ) 345 346 enforced_df = self.enforce_dtypes( 347 df, 348 dtypes=pipe_dtypes, 349 as_polars=as_polars, 350 debug=debug, 351 ) 352 353 if order: 354 enforced_df = _sort_df(enforced_df) 355 return enforced_df
Get a pipe's data from the instance connector.
Parameters
- select_columns (Optional[List[str]], default None):
If provided, only select these given columns.
Otherwise select all available columns (i.e.
SELECT *). - omit_columns (Optional[List[str]], default None): If provided, remove these columns from the selection.
- begin (Union[datetime, int, str, None], default None):
Lower bound datetime to begin searching for data (inclusive).
Translates to a
WHEREclause likeWHERE datetime >= begin. Defaults toNone. - end (Union[datetime, int, str, None], default None):
Upper bound datetime to stop searching for data (inclusive).
Translates to a
WHEREclause likeWHERE datetime < end. Defaults toNone. - params (Optional[Dict[str, Any]], default None):
Filter the retrieved data by a dictionary of parameters.
See
meerschaum.utils.sql.build_wherefor more details. - as_docs (bool, default False):
If
True, return a list of dictionaries rather than a DataFrame. Relies onget_pipe_docsfrom the instance connector if implemented. May be combined withas_chunksto return anIterator[List[Dict]]chunked by time bounds (useful for large result sets without pandas overhead). - as_iterator (bool, default False):
If
True, return a generator of chunks of pipe data. When combined withas_docs=True, yieldsList[Dict]per chunk instead of DataFrames. - as_chunks (bool, default False):
Alias for
as_iterator. When combined withas_docs=True, yieldsList[Dict]per chunk instead of DataFrames. - as_dask (bool, default False):
If
True, return adask.DataFrame(which may be loaded into a Pandas DataFrame withdf.compute()). - as_polars (bool, default False):
If
True, return apolars.DataFrame. When combined withas_iterator=True, yield Polars DataFrames. This may not be combined withas_docsoras_dask. - add_missing_columns (bool, default False):
If
True, add any missing columns fromPipe.dtypesto the dataframe. - chunk_interval (Union[timedelta, int, None], default None):
If
as_iterator, then return chunks withbeginandendseparated by this interval. This may be set underpipe.parameters['chunk_minutes']. By default, use a timedelta of 43200 minutes (30 days). Ifchunk_intervalis an integer and thedatetimeaxis a timestamp, then use a timedelta with the number of minutes configured to this value. If thedatetimeaxis is an integer, anintis used verbatim as the number of epoch units, and atimedeltais scaled into epoch units by the pipe'sprecision. - order (Optional[str], default 'asc'):
If
orderis notNone, sort the resulting dataframe by indices. - limit (Optional[int], default None): If provided, cap the dataframe to this many rows.
- fresh (bool, default False):
If
True, skip local cache and directly query the instance connector. - debug (bool, default False):
Verbosity toggle.
Defaults to
False.
Returns
- A
pd.DataFrameof the pipe's data (default). - A
List[Dict]ifas_docs=True. - An
Iterator[pd.DataFrame]ifas_chunks=True(oras_iterator=True). - An
Iterator[List[Dict]]if bothas_docs=Trueandas_chunks=True. - A
polars.DataFrameifas_polars=True.
453def get_backtrack_data( 454 self, 455 backtrack_minutes: Optional[int] = None, 456 begin: Union[datetime, int, None] = None, 457 params: Optional[Dict[str, Any]] = None, 458 limit: Optional[int] = None, 459 fresh: bool = False, 460 debug: bool = False, 461 **kw: Any 462) -> Optional['pd.DataFrame']: 463 """ 464 Get the most recent data from the instance connector as a Pandas DataFrame. 465 466 Parameters 467 ---------- 468 backtrack_minutes: Optional[int], default None 469 How many minutes from `begin` to select from. 470 If `None`, use `pipe.parameters['fetch']['backtrack_minutes']`. 471 472 begin: Optional[datetime], default None 473 The starting point to search for data. 474 If begin is `None` (default), use the most recent observed datetime 475 (AKA sync_time). 476 477 ``` 478 E.g. begin = 02:00 479 480 Search this region. Ignore this, even if there's data. 481 / / / / / / / / / | 482 -----|----------|----------|----------|----------|----------| 483 00:00 01:00 02:00 03:00 04:00 05:00 484 485 ``` 486 487 params: Optional[Dict[str, Any]], default None 488 The standard Meerschaum `params` query dictionary. 489 490 limit: Optional[int], default None 491 If provided, cap the number of rows to be returned. 492 493 fresh: bool, default False 494 If `True`, Ignore local cache and pull directly from the instance connector. 495 Only comes into effect if a pipe was created with `cache=True`. 496 497 debug: bool default False 498 Verbosity toggle. 499 500 Returns 501 ------- 502 A `pd.DataFrame` for the pipe's data corresponding to the provided parameters. Backtrack data 503 is a convenient way to get a pipe's data "backtracked" from the most recent datetime. 504 """ 505 from meerschaum.utils.venv import Venv 506 from meerschaum.connectors import get_connector_plugin 507 508 if not self.exists(debug=debug): 509 return None 510 511 begin = self.parse_date_bounds(begin, debug=debug) 512 513 backtrack_interval = self.get_backtrack_interval(debug=debug) 514 if backtrack_minutes is None: 515 backtrack_minutes = ( 516 (backtrack_interval.total_seconds() / 60) 517 if isinstance(backtrack_interval, timedelta) 518 else backtrack_interval 519 ) 520 521 if hasattr(self.instance_connector, 'get_backtrack_data'): 522 with Venv(get_connector_plugin(self.instance_connector)): 523 return self.enforce_dtypes( 524 self.instance_connector.get_backtrack_data( 525 pipe=self, 526 begin=begin, 527 backtrack_minutes=backtrack_minutes, 528 params=params, 529 limit=limit, 530 debug=debug, 531 **kw 532 ), 533 debug=debug, 534 ) 535 536 if begin is None: 537 begin = self.get_sync_time(params=params, debug=debug) 538 539 backtrack_interval = ( 540 timedelta(minutes=backtrack_minutes) 541 if isinstance(begin, datetime) 542 else backtrack_minutes 543 ) 544 if begin is not None: 545 begin = begin - backtrack_interval 546 547 kw['order'] = kw.get('order', 'desc') or 'desc' 548 return self.get_data( 549 begin=begin, 550 params=params, 551 debug=debug, 552 limit=limit, 553 **kw 554 )
Get the most recent data from the instance connector as a Pandas DataFrame.
Parameters
- backtrack_minutes (Optional[int], default None):
How many minutes from
beginto select from. IfNone, usepipe.parameters['fetch']['backtrack_minutes']. begin (Optional[datetime], default None): The starting point to search for data. If begin is
None(default), use the most recent observed datetime (AKA sync_time).E.g. begin = 02:00 Search this region. Ignore this, even if there's data. / / / / / / / / / | -----|----------|----------|----------|----------|----------| 00:00 01:00 02:00 03:00 04:00 05:00params (Optional[Dict[str, Any]], default None): The standard Meerschaum
paramsquery dictionary.- limit (Optional[int], default None): If provided, cap the number of rows to be returned.
- fresh (bool, default False):
If
True, Ignore local cache and pull directly from the instance connector. Only comes into effect if a pipe was created withcache=True. - debug (bool default False): Verbosity toggle.
Returns
- A
pd.DataFramefor the pipe's data corresponding to the provided parameters. Backtrack data - is a convenient way to get a pipe's data "backtracked" from the most recent datetime.
557def get_rowcount( 558 self, 559 begin: Union[datetime, int, None] = None, 560 end: Union[datetime, int, None] = None, 561 params: Optional[Dict[str, Any]] = None, 562 remote: bool = False, 563 debug: bool = False 564) -> int: 565 """ 566 Get a Pipe's instance or remote rowcount. 567 568 Parameters 569 ---------- 570 begin: Optional[datetime], default None 571 Count rows where datetime > begin. 572 573 end: Optional[datetime], default None 574 Count rows where datetime < end. 575 576 remote: bool, default False 577 Count rows from a pipe's remote source. 578 **NOTE**: This is experimental! 579 580 debug: bool, default False 581 Verbosity toggle. 582 583 Returns 584 ------- 585 An `int` of the number of rows in the pipe corresponding to the provided parameters. 586 Returned 0 if the pipe does not exist. 587 """ 588 from meerschaum.utils.warnings import warn 589 from meerschaum.utils.venv import Venv 590 from meerschaum.connectors import get_connector_plugin 591 from meerschaum.utils.misc import filter_keywords 592 593 begin, end = self.parse_date_bounds(begin, end, debug=debug) 594 connector = self.instance_connector if not remote else self.connector 595 try: 596 with Venv(get_connector_plugin(connector)): 597 if not hasattr(connector, 'get_pipe_rowcount'): 598 warn( 599 f"Connectors of type '{connector.type}' " 600 "do not implement `get_pipe_rowcount()`.", 601 stack=False, 602 ) 603 return 0 604 kwargs = filter_keywords( 605 connector.get_pipe_rowcount, 606 begin=begin, 607 end=end, 608 params=params, 609 remote=remote, 610 debug=debug, 611 ) 612 if remote and 'remote' not in kwargs: 613 warn( 614 f"Connectors of type '{connector.type}' do not support remote rowcounts.", 615 stack=False, 616 ) 617 return 0 618 rowcount = connector.get_pipe_rowcount( 619 self, 620 begin=begin, 621 end=end, 622 params=params, 623 remote=remote, 624 debug=debug, 625 ) 626 if rowcount is None: 627 return 0 628 return rowcount 629 except AttributeError as e: 630 warn(e) 631 if remote: 632 return 0 633 warn(f"Failed to get a rowcount for {self}.") 634 return 0
Get a Pipe's instance or remote rowcount.
Parameters
- begin (Optional[datetime], default None): Count rows where datetime > begin.
- end (Optional[datetime], default None): Count rows where datetime < end.
- remote (bool, default False): Count rows from a pipe's remote source. NOTE: This is experimental!
- debug (bool, default False): Verbosity toggle.
Returns
- An
intof the number of rows in the pipe corresponding to the provided parameters. - Returned 0 if the pipe does not exist.
637def get_size( 638 self, 639 debug: bool = False, 640 **kw: Any 641) -> Union[int, None]: 642 """ 643 Return the on-disk size of the pipe's target table in bytes. 644 645 Parameters 646 ---------- 647 debug: bool, default False 648 Verbosity toggle. 649 650 Returns 651 ------- 652 An `int` of the number of bytes occupied by the pipe's target table, 653 or `None` if the size could not be determined (e.g. the connector does 654 not implement `get_pipe_size()` or the table does not exist). 655 """ 656 from meerschaum.utils.warnings import warn 657 from meerschaum.utils.venv import Venv 658 from meerschaum.connectors import get_connector_plugin 659 from meerschaum.utils.misc import filter_keywords 660 661 connector = self.instance_connector 662 try: 663 with Venv(get_connector_plugin(connector)): 664 if not hasattr(connector, 'get_pipe_size'): 665 return None 666 kwargs = filter_keywords( 667 connector.get_pipe_size, 668 debug=debug, 669 **kw 670 ) 671 return connector.get_pipe_size(self, **kwargs) 672 except NotImplementedError: 673 return None 674 except Exception as e: 675 warn(f"Failed to get the size of {self}:\n{e}", stack=False) 676 return None
Return the on-disk size of the pipe's target table in bytes.
Parameters
- debug (bool, default False): Verbosity toggle.
Returns
- An
intof the number of bytes occupied by the pipe's target table, - or
Noneif the size could not be determined (e.g. the connector does - not implement
get_pipe_size()or the table does not exist).
1060def get_doc(self, **kwargs) -> Union[Dict[str, Any], None]: 1061 """ 1062 Convenience function to return a single row as a dictionary (or `None`) from `Pipe.get_data(). 1063 Keywords arguments are passed to `Pipe.get_data()`. 1064 """ 1065 from meerschaum.utils.warnings import warn 1066 kwargs['limit'] = 1 1067 kwargs['as_docs'] = True 1068 try: 1069 docs = self.get_data(**kwargs) 1070 if not docs: 1071 return None 1072 return docs[0] 1073 except Exception as e: 1074 warn(f"Failed to read value from {self}:\n{e}", stack=False) 1075 return None
Convenience function to return a single row as a dictionary (or None) from Pipe.get_data().
Keywords arguments are passed toPipe.get_data()`.
1077def get_docs(self, **kwargs) -> list[dict[str, Any]]: 1078 """ 1079 Convenience method to return a pipe's data as a list of dictionaries. 1080 Relies on `get_pipe_docs` from the instance connector if implemented. 1081 """ 1082 kwargs['as_docs'] = True 1083 return self.get_data(**kwargs)
Convenience method to return a pipe's data as a list of dictionaries.
Relies on get_pipe_docs from the instance connector if implemented.
1085def get_value( 1086 self, 1087 column: str, 1088 params: Optional[Dict[str, Any]] = None, 1089 **kwargs: Any 1090) -> Any: 1091 """ 1092 Convenience function to return a single value (or `None`) from `Pipe.get_data()`. 1093 Keywords arguments are passed to `Pipe.get_data()`. 1094 """ 1095 from meerschaum.utils.warnings import warn 1096 kwargs['select_columns'] = [column] 1097 kwargs['limit'] = 1 1098 kwargs['as_docs'] = True 1099 try: 1100 docs = self.get_data(params=params, **kwargs) 1101 if not docs: 1102 return None 1103 if column not in docs[0]: 1104 raise ValueError(f"Column '{column}' was not included in the result set.") 1105 return docs[0][column] 1106 except Exception as e: 1107 warn(f"Failed to read value from {self}:\n{e}", stack=False) 1108 return None
Convenience function to return a single value (or None) from Pipe.get_data().
Keywords arguments are passed to Pipe.get_data().
679def get_chunk_interval( 680 self, 681 chunk_interval: Union[timedelta, int, None] = None, 682 debug: bool = False, 683) -> Union[timedelta, int]: 684 """ 685 Get the chunk interval to use for this pipe. 686 687 The size is read from the `verify` parameters. Any one of these aliased keys may be used 688 (the first present, in this priority order, wins): 689 690 - `verify.chunk_minutes` (the default; 43200 — 30 days — if none is set) 691 - `verify.chunk_hours` 692 - `verify.chunk_days` 693 - `verify.chunk_weeks` 694 - `verify.chunk_years` 695 - `verify.chunk_seconds` 696 697 For an integer datetime axis, `verify.chunk_range` (if set) is used verbatim as the chunk size 698 in epoch units. Otherwise the time-based size above is converted to epoch units via the pipe's 699 `precision`, or — preserving legacy behavior when no `precision` is set — its minutes are used 700 verbatim. 701 702 Parameters 703 ---------- 704 chunk_interval: Union[timedelta, int, None], default None 705 If provided, use this value instead of the `verify` keys. 706 An `int` is already in the axis's own units — epoch units for an integer axis, 707 minutes otherwise — and is used verbatim. A `timedelta` is a duration, so on an 708 integer axis it is scaled into epoch units by the pipe's `precision`. 709 710 Returns 711 ------- 712 The chunk interval (`timedelta` or `int`) to use with this pipe's `datetime` axis. 713 """ 714 dt_col = self.columns.get('datetime', None) 715 dt_dtype = self.dtypes.get(dt_col, 'datetime') if dt_col is not None else 'datetime' 716 is_int_axis = 'int' in str(dt_dtype).lower() 717 verify_params = self.parameters.get('verify', {}) 718 719 ### An explicit `chunk_interval` argument overrides the `verify` keys. 720 if chunk_interval is not None: 721 if dt_col is not None and is_int_axis: 722 ### An `int` is already in epoch units; a duration must be scaled like the 723 ### no-argument path below. 724 if isinstance(chunk_interval, int): 725 return chunk_interval 726 return self._scale_delta_to_int_axis(chunk_interval) 727 728 chunk_minutes = ( 729 chunk_interval 730 if isinstance(chunk_interval, int) 731 else int(chunk_interval.total_seconds() / 60) 732 ) 733 return timedelta(minutes=chunk_minutes) 734 735 ### Integer axis: an explicit `verify.chunk_range` is the chunk size in epoch units, verbatim. 736 if dt_col is not None and is_int_axis: 737 chunk_range = verify_params.get('chunk_range', None) 738 if chunk_range is not None: 739 return int(chunk_range) 740 741 ### Resolve the time-based chunk size from the aliased `verify.chunk_*` keys (priority order 742 ### matches the `bound_*` aliases). Falls back to the configured `chunk_minutes` default. 743 chunk_delta = None 744 for suffix in ('minutes', 'hours', 'days', 'weeks', 'years', 'seconds'): 745 val = verify_params.get('chunk_' + suffix, None) 746 if val is None: 747 continue 748 ### `timedelta` has no `years` kwarg; approximate a year as 365 days. 749 chunk_delta = timedelta(days=(val * 365)) if suffix == 'years' else timedelta(**{suffix: val}) 750 break 751 if chunk_delta is None: 752 default_chunk_minutes = get_config('pipes', 'parameters', 'verify', 'chunk_minutes') 753 chunk_delta = timedelta(minutes=default_chunk_minutes) 754 755 if dt_col is None: 756 return chunk_delta 757 758 if is_int_axis: 759 return self._scale_delta_to_int_axis(chunk_delta) 760 761 return chunk_delta
Get the chunk interval to use for this pipe.
The size is read from the verify parameters. Any one of these aliased keys may be used
(the first present, in this priority order, wins):
- `verify.chunk_minutes` (the default; 43200 — 30 days — if none is set)
- `verify.chunk_hours`
- `verify.chunk_days`
- `verify.chunk_weeks`
- `verify.chunk_years`
- `verify.chunk_seconds`
For an integer datetime axis, verify.chunk_range (if set) is used verbatim as the chunk size
in epoch units. Otherwise the time-based size above is converted to epoch units via the pipe's
precision, or — preserving legacy behavior when no precision is set — its minutes are used
verbatim.
Parameters
- chunk_interval (Union[timedelta, int, None], default None):
If provided, use this value instead of the
verifykeys. Anintis already in the axis's own units — epoch units for an integer axis, minutes otherwise — and is used verbatim. Atimedeltais a duration, so on an integer axis it is scaled into epoch units by the pipe'sprecision.
Returns
- The chunk interval (
timedeltaorint) to use with this pipe'sdatetimeaxis.
785def get_chunk_bounds( 786 self, 787 begin: Union[datetime, int, None] = None, 788 end: Union[datetime, int, None] = None, 789 bounded: bool = False, 790 chunk_interval: Union[timedelta, int, None] = None, 791 align: bool = False, 792 debug: bool = False, 793) -> List[ 794 Tuple[ 795 Union[datetime, int, None], 796 Union[datetime, int, None], 797 ] 798]: 799 """ 800 Return a list of datetime bounds for iterating over the pipe's `datetime` axis. 801 802 Parameters 803 ---------- 804 begin: Union[datetime, int, None], default None 805 If provided, do not select less than this value. 806 Otherwise the first chunk will be unbounded. 807 808 end: Union[datetime, int, None], default None 809 If provided, do not select greater than or equal to this value. 810 Otherwise the last chunk will be unbounded. 811 812 bounded: bool, default False 813 If `True`, do not include `None` in the first chunk. 814 815 chunk_interval: Union[timedelta, int, None], default None 816 If provided, use this interval for the size of chunk boundaries. 817 The default value for this pipe may be set 818 under `pipe.parameters['verify']['chunk_minutes']`. 819 820 align: bool, default False 821 If `True`, anchor the interior chunk boundaries to a fixed Unix-epoch grid (the same 822 grid used for native range partitioning) rather than to `begin`. This makes the 823 boundaries deterministic across re-syncs and aligned with the pipe's partitions 824 (used by `Pipe.verify()`). The first chunk's lower bound and the last chunk's upper 825 bound are still clamped to `begin` / `end`. 826 827 debug: bool, default False 828 Verbosity toggle. 829 830 Returns 831 ------- 832 A list of chunk bounds (datetimes or integers). 833 If unbounded, the first and last chunks will include `None`. 834 """ 835 from datetime import timedelta 836 from meerschaum.utils.dtypes import are_dtypes_equal 837 from meerschaum.utils.misc import interval_str 838 include_less_than_begin = not bounded and begin is None 839 include_greater_than_end = not bounded and end is None 840 if begin is None: 841 begin = self.get_sync_time(newest=False, debug=debug) 842 consolidate_end_chunk = False 843 if end is None: 844 end = self.get_sync_time(newest=True, debug=debug) 845 if end is not None and hasattr(end, 'tzinfo'): 846 end += timedelta(minutes=1) 847 consolidate_end_chunk = True 848 elif are_dtypes_equal(str(type(end)), 'int'): 849 end += 1 850 consolidate_end_chunk = True 851 852 if begin is None and end is None: 853 return [(None, None)] 854 855 begin, end = self.parse_date_bounds(begin, end, debug=debug) 856 857 if begin and end: 858 if begin >= end: 859 return ( 860 [(begin, begin)] 861 if bounded 862 else [(begin, None)] 863 ) 864 if end <= begin: 865 return ( 866 [(end, end)] 867 if bounded 868 else [(None, begin)] 869 ) 870 871 ### Set the chunk interval under `pipe.parameters['verify']['chunk_minutes']`. 872 chunk_interval = self.get_chunk_interval(chunk_interval, debug=debug) 873 874 ### Anchor the interior boundaries to a fixed Unix-epoch grid (matching native range 875 ### partitioning, see `SQLConnector._partition_bounds`) so chunk edges line up with partition 876 ### edges and stay deterministic regardless of `begin`. The first chunk is clamped back to 877 ### `begin` below. 878 begin_cursor = begin 879 if align and begin is not None: 880 if isinstance(chunk_interval, int): 881 begin_cursor = (int(begin) // chunk_interval) * chunk_interval 882 else: 883 epoch = ( 884 datetime(1970, 1, 1, tzinfo=begin.tzinfo) 885 if getattr(begin, 'tzinfo', None) is not None 886 else datetime(1970, 1, 1) 887 ) 888 n = (begin - epoch) // chunk_interval 889 begin_cursor = epoch + (n * chunk_interval) 890 891 ### Build a list of tuples containing the chunk boundaries 892 ### so that we can sync multiple chunks in parallel. 893 ### Run `verify pipes --workers 1` to sync chunks in series. 894 chunk_bounds = [] 895 num_chunks = 0 896 max_chunks = 1_000_000 897 while begin_cursor < end: 898 end_cursor = begin_cursor + chunk_interval 899 chunk_bounds.append((begin_cursor, end_cursor)) 900 begin_cursor = end_cursor 901 num_chunks += 1 902 if num_chunks >= max_chunks: 903 raise ValueError( 904 f"Too many chunks of size '{interval_str(chunk_interval)}' " 905 f"between '{begin}' and '{end}'." 906 ) 907 908 if num_chunks > 1 and consolidate_end_chunk: 909 last_bounds, second_last_bounds = chunk_bounds[-1], chunk_bounds[-2] 910 chunk_bounds = chunk_bounds[:-2] 911 chunk_bounds.append((second_last_bounds[0], last_bounds[1])) 912 913 ### The chunk interval might be too large. 914 if not chunk_bounds and end >= begin: 915 chunk_bounds = [(begin, end)] 916 917 ### Truncate the last chunk to the end timestamp. 918 if chunk_bounds[-1][1] > end: 919 chunk_bounds[-1] = (chunk_bounds[-1][0], end) 920 921 ### Pop the last chunk if its bounds are equal. 922 if chunk_bounds[-1][0] == chunk_bounds[-1][1]: 923 chunk_bounds = chunk_bounds[:-1] 924 925 ### Clamp the epoch-aligned first chunk's lower bound back to the requested `begin` so the 926 ### returned range still starts exactly at `begin` (only the interior edges are grid-aligned). 927 if ( 928 align 929 and chunk_bounds 930 and chunk_bounds[0][0] is not None 931 and chunk_bounds[0][0] < begin 932 ): 933 chunk_bounds[0] = (begin, chunk_bounds[0][1]) 934 935 if include_less_than_begin: 936 chunk_bounds = [(None, begin)] + chunk_bounds 937 if include_greater_than_end: 938 chunk_bounds = chunk_bounds + [(end, None)] 939 940 return chunk_bounds
Return a list of datetime bounds for iterating over the pipe's datetime axis.
Parameters
- begin (Union[datetime, int, None], default None): If provided, do not select less than this value. Otherwise the first chunk will be unbounded.
- end (Union[datetime, int, None], default None): If provided, do not select greater than or equal to this value. Otherwise the last chunk will be unbounded.
- bounded (bool, default False):
If
True, do not includeNonein the first chunk. - chunk_interval (Union[timedelta, int, None], default None):
If provided, use this interval for the size of chunk boundaries.
The default value for this pipe may be set
under
pipe.parameters['verify']['chunk_minutes']. - align (bool, default False):
If
True, anchor the interior chunk boundaries to a fixed Unix-epoch grid (the same grid used for native range partitioning) rather than tobegin. This makes the boundaries deterministic across re-syncs and aligned with the pipe's partitions (used byPipe.verify()). The first chunk's lower bound and the last chunk's upper bound are still clamped tobegin/end. - debug (bool, default False): Verbosity toggle.
Returns
- A list of chunk bounds (datetimes or integers).
- If unbounded, the first and last chunks will include
None.
943def get_chunk_bounds_batches( 944 self, 945 chunk_bounds: List[Tuple[Union[datetime, int, None], Union[datetime, int, None]]], 946 batchsize: Optional[int] = None, 947 workers: Optional[int] = None, 948 debug: bool = False, 949) -> List[ 950 Tuple[ 951 Tuple[ 952 Union[datetime, int, None], 953 Union[datetime, int, None], 954 ], ... 955 ] 956]: 957 """ 958 Return a list of tuples of chunk bounds of size `batchsize`. 959 960 Parameters 961 ---------- 962 chunk_bounds: List[Tuple[Union[datetime, int, None], Union[datetime, int, None]]] 963 A list of chunk_bounds (see `Pipe.get_chunk_bounds()`). 964 965 batchsize: Optional[int], default None 966 How many chunks to include in a batch. Defaults to `Pipe.get_num_workers()`. 967 968 workers: Optional[int], default None 969 If `batchsize` is `None`, use this as the desired number of workers. 970 Passed to `Pipe.get_num_workers()`. 971 972 Returns 973 ------- 974 A list of tuples of chunk bound tuples. 975 """ 976 from meerschaum.utils.misc import iterate_chunks 977 978 if batchsize is None: 979 batchsize = self.get_num_workers(workers=workers) 980 981 return [ 982 tuple( 983 _batch_chunk_bounds 984 for _batch_chunk_bounds in batch 985 if _batch_chunk_bounds is not None 986 ) 987 for batch in iterate_chunks(chunk_bounds, batchsize) 988 if batch 989 ]
Return a list of tuples of chunk bounds of size batchsize.
Parameters
- chunk_bounds (List[Tuple[Union[datetime, int, None], Union[datetime, int, None]]]):
A list of chunk_bounds (see
Pipe.get_chunk_bounds()). - batchsize (Optional[int], default None):
How many chunks to include in a batch. Defaults to
Pipe.get_num_workers(). - workers (Optional[int], default None):
If
batchsizeisNone, use this as the desired number of workers. Passed toPipe.get_num_workers().
Returns
- A list of tuples of chunk bound tuples.
992def parse_date_bounds(self, *dt_vals: Union[datetime, int, None], debug: bool = False) -> Union[ 993 datetime, 994 int, 995 str, 996 None, 997 Tuple[Union[datetime, int, str, None]] 998]: 999 """ 1000 Given a date bound (begin, end), coerce a timezone if necessary. 1001 """ 1002 from meerschaum.utils.misc import is_int 1003 from meerschaum.utils.dtypes import coerce_timezone, MRSM_PD_DTYPES, are_dtypes_equal 1004 from meerschaum.utils.warnings import warn 1005 dateutil_parser = mrsm.attempt_import('dateutil.parser') 1006 1007 _columns = None 1008 _dtypes = None 1009 1010 def _get_coercion_info(): 1011 nonlocal _columns, _dtypes 1012 if _columns is None: 1013 _columns = self.get_parameters(debug=debug).get('columns', {}) or {} 1014 if _dtypes is None: 1015 _dtypes = self.get_dtypes(debug=debug) 1016 1017 def _parse_date_bound(dt_val): 1018 if dt_val is None: 1019 return None 1020 1021 if isinstance(dt_val, int): 1022 return dt_val 1023 1024 if dt_val == '': 1025 return '' 1026 1027 if is_int(dt_val): 1028 return int(dt_val) 1029 1030 if isinstance(dt_val, str): 1031 try: 1032 dt_val = dateutil_parser.parse(dt_val) 1033 except Exception as e: 1034 warn(f"Could not parse '{dt_val}' as datetime:\n{e}") 1035 return None 1036 1037 _get_coercion_info() 1038 dt_col = _columns.get('datetime', None) 1039 dt_typ = str(_dtypes.get(dt_col, 'datetime')) 1040 if are_dtypes_equal(dt_typ, 'int'): 1041 if self.get_parameters(debug=debug).get('precision'): 1042 from meerschaum.utils.dtypes import datetime_to_int 1043 return datetime_to_int(dt_val, self.precision['unit']) 1044 from meerschaum.utils.warnings import error 1045 error( 1046 f"Cannot use datetime bound '{dt_val}' on the non-epoch integer axis " 1047 f"of {self}.\n Pass an integer instead, or set the `precision` parameter.", 1048 ValueError, 1049 ) 1050 if dt_typ == 'datetime': 1051 dt_typ = MRSM_PD_DTYPES['datetime'] 1052 return coerce_timezone(dt_val, strip_utc=('utc' not in dt_typ.lower())) 1053 1054 bounds = tuple(_parse_date_bound(dt_val) for dt_val in dt_vals) 1055 if len(bounds) == 1: 1056 return bounds[0] 1057 return bounds
Given a date bound (begin, end), coerce a timezone if necessary.
12def register( 13 self, 14 debug: bool = False, 15 **kw: Any 16) -> SuccessTuple: 17 """ 18 Register a new Pipe along with its attributes. 19 20 Parameters 21 ---------- 22 debug: bool, default False 23 Verbosity toggle. 24 25 kw: Any 26 Keyword arguments to pass to `instance_connector.register_pipe()`. 27 28 Returns 29 ------- 30 A `SuccessTuple` of success, message. 31 """ 32 if self.temporary: 33 return False, "Cannot register pipes created with `temporary=True` (read-only)." 34 35 from meerschaum.utils.formatting import get_console 36 from meerschaum.utils.venv import Venv 37 from meerschaum.connectors import get_connector_plugin, custom_types 38 from meerschaum.config._patch import apply_patch_to_config 39 40 import warnings 41 with warnings.catch_warnings(): 42 warnings.simplefilter('ignore') 43 try: 44 _conn = self.connector 45 except Exception: 46 _conn = None 47 48 if isinstance(_conn, str): 49 _conn = None 50 51 if ( 52 _conn is not None 53 and 54 (_conn.type == 'plugin' or _conn.type in custom_types) 55 and 56 getattr(_conn, 'register', None) is not None 57 ): 58 try: 59 with Venv(get_connector_plugin(_conn), debug=debug): 60 params = self.connector.register(self) 61 except Exception: 62 get_console().print_exception() 63 params = None 64 params = {} if params is None else params 65 if not isinstance(params, dict): 66 from meerschaum.utils.warnings import warn 67 warn( 68 f"Invalid parameters returned from `register()` in connector {self.connector}:\n" 69 + f"{params}" 70 ) 71 else: 72 self.parameters = apply_patch_to_config(params, self.parameters) 73 74 if not self.parameters: 75 cols = self.columns if self.columns else {'datetime': None, 'id': None} 76 self.parameters = { 77 'columns': cols, 78 } 79 80 ### Refuse an index column which cannot be sorted or deduplicated, rather than 81 ### registering a pipe whose second sync is guaranteed to fail. 82 indices_success, indices_msg = self.validate_indices(debug=debug) 83 if not indices_success: 84 return indices_success, indices_msg 85 86 with Venv(get_connector_plugin(self.instance_connector)): 87 return self.instance_connector.register_pipe(self, debug=debug, **kw)
Register a new Pipe along with its attributes.
Parameters
- debug (bool, default False): Verbosity toggle.
- kw (Any):
Keyword arguments to pass to
instance_connector.register_pipe().
Returns
- A
SuccessTupleof success, message.
20@property 21def attributes(self) -> Dict[str, Any]: 22 """ 23 Return a dictionary of a pipe's keys and parameters. 24 These values are reflected directly from the pipes table of the instance. 25 """ 26 from meerschaum.config import get_config 27 from meerschaum.config._patch import apply_patch_to_config 28 from meerschaum.utils.venv import Venv 29 from meerschaum.connectors import get_connector_plugin 30 from meerschaum.utils.dtypes import get_current_timestamp 31 32 if self.temporary: 33 return self._attributes 34 35 with self._attributes_lock: 36 # Instance connectors may read local parameters while fetching metadata. 37 if self.__dict__.get('_attributes_loading', False): 38 return self._attributes 39 timeout_seconds = get_config('pipes', 'attributes', 'local_cache_timeout_seconds') 40 now = get_current_timestamp('ms', as_int=True) / 1000 41 sync_time = self._get_cached_value('_attributes_sync_time', debug=self.debug) 42 timed_out = ( 43 sync_time is None 44 or (timeout_seconds is not None and (now - sync_time) >= timeout_seconds) 45 ) 46 if timed_out: 47 local_attributes = self._get_cached_value('attributes', debug=self.debug) or {} 48 self._attributes_loading = True 49 try: 50 with Venv(get_connector_plugin(self.instance_connector)): 51 instance_attributes = self.instance_connector.get_pipe_attributes(self) 52 if not isinstance(instance_attributes, dict) or ( 53 not instance_attributes and self._get_cached_value('_id') is not None 54 ): 55 raise RuntimeError(f"Missing metadata for registered pipe {self}.") 56 self._cache_value( 57 'attributes', 58 apply_patch_to_config(instance_attributes, local_attributes), 59 memory_only=True, 60 debug=self.debug, 61 ) 62 except Exception as e: 63 self._attributes_load_failures = self.__dict__.get('_attributes_load_failures', 0) + 1 64 self._clear_cache_key('_attributes_sync_time', debug=self.debug) 65 warn(f"Failed to load attributes for {self}: {e}") 66 return self._attributes 67 finally: 68 self._attributes_loading = False 69 self._cache_value( 70 '_attributes_sync_time', get_current_timestamp('ms', as_int=True) / 1000, 71 memory_only=True, debug=self.debug, 72 ) 73 74 return self._attributes
Return a dictionary of a pipe's keys and parameters. These values are reflected directly from the pipes table of the instance.
202@property 203def parameters(self) -> Optional[Dict[str, Any]]: 204 """ 205 Return the parameters dictionary of the pipe. 206 """ 207 return self.get_parameters(debug=self.debug)
Return the parameters dictionary of the pipe.
219@property 220def columns(self) -> Union[Dict[str, str], None]: 221 """ 222 Return the `columns` dictionary defined in `meerschaum.Pipe.parameters`. 223 """ 224 cols = self.parameters.get('columns', {}) 225 if not isinstance(cols, dict): 226 return {} 227 return {col_ix: col for col_ix, col in cols.items() if col and col_ix}
Return the columns dictionary defined in meerschaum.Pipe.parameters.
244@property 245def indices(self) -> Union[Dict[str, Union[str, List[str]]], None]: 246 """ 247 Return the `indices` dictionary defined in `meerschaum.Pipe.parameters`. 248 """ 249 _parameters = self.get_parameters(debug=self.debug) 250 indices_key = ( 251 'indexes' 252 if 'indexes' in _parameters 253 else 'indices' 254 ) 255 256 _indices = _parameters.get(indices_key, {}) 257 _columns = self.columns 258 dt_col = _columns.get('datetime', None) 259 if not isinstance(_indices, dict): 260 _indices = {} 261 unique_cols = list(set(( 262 [dt_col] 263 if dt_col 264 else [] 265 ) + [ 266 col 267 for col_ix, col in _columns.items() 268 if col and col_ix != 'datetime' 269 ])) 270 return { 271 **({'unique': unique_cols} if len(unique_cols) > 1 else {}), 272 **{col_ix: col for col_ix, col in _columns.items() if col}, 273 **_indices 274 }
Return the indices dictionary defined in meerschaum.Pipe.parameters.
277@property 278def indexes(self) -> Union[Dict[str, Union[str, List[str]]], None]: 279 """ 280 Alias for `meerschaum.Pipe.indices`. 281 """ 282 return self.indices
Alias for meerschaum.Pipe.indices.
333@property 334def dtypes(self) -> Dict[str, Any]: 335 """ 336 If defined, return the `dtypes` dictionary defined in `meerschaum.Pipe.parameters`. 337 """ 338 return self.get_dtypes(refresh=False, debug=self.debug)
If defined, return the dtypes dictionary defined in meerschaum.Pipe.parameters.
491@property 492def autoincrement(self) -> bool: 493 """ 494 Return the `autoincrement` parameter for the pipe. 495 """ 496 return self.parameters.get('autoincrement', False)
Return the autoincrement parameter for the pipe.
507@property 508def autotime(self) -> bool: 509 """ 510 Return the `autotime` parameter for the pipe. 511 """ 512 return self.parameters.get('autotime', False)
Return the autotime parameter for the pipe.
458@property 459def upsert(self) -> bool: 460 """ 461 Return whether `upsert` is set for the pipe. 462 """ 463 return self.parameters.get('upsert', False)
Return whether upsert is set for the pipe.
474@property 475def static(self) -> bool: 476 """ 477 Return whether `static` is set for the pipe. 478 """ 479 return self.parameters.get('static', False)
Return whether static is set for the pipe.
523@property 524def tzinfo(self) -> Union[None, timezone]: 525 """ 526 Return `timezone.utc` if the pipe is timezone-aware. 527 """ 528 _tzinfo = self._get_cached_value('tzinfo', debug=self.debug) 529 if _tzinfo is not None: 530 return _tzinfo if _tzinfo != 'None' else None 531 532 _tzinfo = None 533 dt_col = self.columns.get('datetime', None) 534 dt_typ = str(self.dtypes.get(dt_col, 'datetime')) if dt_col else None 535 if self.autotime: 536 ts_col = mrsm.get_config('pipes', 'autotime', 'column_name_if_datetime_missing') 537 ts_typ = self.dtypes.get(ts_col, 'datetime') 538 dt_typ = ts_typ 539 540 if dt_typ and 'utc' in dt_typ.lower() or dt_typ == 'datetime': 541 _tzinfo = timezone.utc 542 543 self._cache_value('tzinfo', (_tzinfo if _tzinfo is not None else 'None'), debug=self.debug) 544 return _tzinfo
Return timezone.utc if the pipe is timezone-aware.
547@property 548def enforce(self) -> bool: 549 """ 550 Return the `enforce` parameter for the pipe. 551 """ 552 return self.parameters.get('enforce', True)
Return the enforce parameter for the pipe.
563@property 564def null_indices(self) -> bool: 565 """ 566 Return the `null_indices` parameter for the pipe. 567 """ 568 return self.parameters.get('null_indices', True)
Return the null_indices parameter for the pipe.
579@property 580def mixed_numerics(self) -> bool: 581 """ 582 Return the `mixed_numerics` parameter for the pipe. 583 """ 584 return self.parameters.get('mixed_numerics', True)
Return the mixed_numerics parameter for the pipe.
595def get_columns(self, *args: str, error: bool = False) -> Union[str, Tuple[str]]: 596 """ 597 Check if the requested columns are defined. 598 599 Parameters 600 ---------- 601 *args: str 602 The column names to be retrieved. 603 604 error: bool, default False 605 If `True`, raise an `Exception` if the specified column is not defined. 606 607 Returns 608 ------- 609 A tuple of the same size of `args` or a `str` if `args` is a single argument. 610 611 Examples 612 -------- 613 >>> pipe = mrsm.Pipe('test', 'test') 614 >>> pipe.columns = {'datetime': 'dt', 'id': 'id'} 615 >>> pipe.get_columns('datetime', 'id') 616 ('dt', 'id') 617 >>> pipe.get_columns('value', error=True) 618 Exception: 🛑 Missing 'value' column for Pipe('test', 'test'). 619 """ 620 from meerschaum.utils.warnings import error as _error 621 if not args: 622 args = tuple(self.columns.keys()) 623 col_names = [] 624 for col in args: 625 col_name = None 626 try: 627 col_name = self.columns[col] 628 if col_name is None and error: 629 _error(f"Please define the name of the '{col}' column for {self}.") 630 except Exception: 631 col_name = None 632 if col_name is None and error: 633 _error(f"Missing '{col}'" + f" column for {self}.") 634 col_names.append(col_name) 635 if len(col_names) == 1: 636 return col_names[0] 637 return tuple(col_names)
Check if the requested columns are defined.
Parameters
- *args (str): The column names to be retrieved.
- error (bool, default False):
If
True, raise anExceptionif the specified column is not defined.
Returns
- A tuple of the same size of
argsor astrifargsis a single argument.
Examples
>>> pipe = mrsm.Pipe('test', 'test')
>>> pipe.columns = {'datetime': 'dt', 'id': 'id'}
>>> pipe.get_columns('datetime', 'id')
('dt', 'id')
>>> pipe.get_columns('value', error=True)
Exception: 🛑 Missing 'value' column for Pipe('test', 'test').
640def get_columns_types( 641 self, 642 refresh: bool = False, 643 debug: bool = False, 644) -> Union[Dict[str, str], None]: 645 """ 646 Get a dictionary of a pipe's column names and their types. 647 648 Parameters 649 ---------- 650 refresh: bool, default False 651 If `True`, invalidate the cache and fetch directly from the instance connector. 652 653 debug: bool, default False: 654 Verbosity toggle. 655 656 Returns 657 ------- 658 A dictionary of column names (`str`) to column types (`str`). 659 660 Examples 661 -------- 662 >>> pipe.get_columns_types() 663 { 664 'dt': 'TIMESTAMP WITH TIMEZONE', 665 'id': 'BIGINT', 666 'val': 'DOUBLE PRECISION', 667 } 668 >>> 669 """ 670 from meerschaum.connectors import get_connector_plugin 671 from meerschaum.utils.dtypes import get_current_timestamp 672 673 now = get_current_timestamp('ms', as_int=True) / 1000 674 cache_seconds = ( 675 mrsm.get_config('pipes', 'static', 'static_schema_cache_seconds') 676 if self.static 677 else mrsm.get_config('pipes', 'dtypes', 'columns_types_cache_seconds') 678 ) 679 if refresh: 680 self._clear_cache_key('_columns_types_timestamp', debug=debug) 681 self._clear_cache_key('_columns_types', debug=debug) 682 683 _columns_types = self._get_cached_value('_columns_types', debug=debug) 684 if _columns_types: 685 columns_types_timestamp = self._get_cached_value('_columns_types_timestamp', debug=debug) 686 if columns_types_timestamp is not None: 687 delta = now - columns_types_timestamp 688 if delta < cache_seconds: 689 if debug: 690 dprint( 691 f"Returning cached `columns_types` for {self} " 692 f"({round(delta, 2)} seconds old)." 693 ) 694 return _columns_types 695 696 with mrsm.Venv(get_connector_plugin(self.instance_connector)): 697 _columns_types = ( 698 self.instance_connector.get_pipe_columns_types(self, debug=debug) 699 if hasattr(self.instance_connector, 'get_pipe_columns_types') 700 else None 701 ) 702 703 self._cache_value('_columns_types', _columns_types, debug=debug) 704 self._cache_value('_columns_types_timestamp', now, debug=debug) 705 return _columns_types or {}
Get a dictionary of a pipe's column names and their types.
Parameters
- refresh (bool, default False):
If
True, invalidate the cache and fetch directly from the instance connector. - debug (bool, default False:): Verbosity toggle.
Returns
- A dictionary of column names (
str) to column types (str).
Examples
>>> pipe.get_columns_types()
{
'dt': 'TIMESTAMP WITH TIMEZONE',
'id': 'BIGINT',
'val': 'DOUBLE PRECISION',
}
>>>
708def get_columns_indices( 709 self, 710 debug: bool = False, 711 refresh: bool = False, 712) -> Dict[str, List[Dict[str, str]]]: 713 """ 714 Return a dictionary mapping columns to index information. 715 """ 716 from meerschaum.connectors import get_connector_plugin 717 from meerschaum.utils.dtypes import get_current_timestamp 718 719 now = get_current_timestamp('ms', as_int=True) / 1000 720 cache_seconds = ( 721 mrsm.get_config('pipes', 'static', 'static_schema_cache_seconds') 722 if self.static 723 else mrsm.get_config('pipes', 'dtypes', 'columns_types_cache_seconds') 724 ) 725 if refresh: 726 self._clear_cache_key('_columns_indices_timestamp', debug=debug) 727 self._clear_cache_key('_columns_indices', debug=debug) 728 729 _columns_indices = self._get_cached_value('_columns_indices', debug=debug) 730 731 if _columns_indices: 732 columns_indices_timestamp = self._get_cached_value('_columns_indices_timestamp', debug=debug) 733 if columns_indices_timestamp is not None: 734 delta = now - columns_indices_timestamp 735 if delta < cache_seconds: 736 if debug: 737 dprint( 738 f"Returning cached `columns_indices` for {self} " 739 f"({round(delta, 2)} seconds old)." 740 ) 741 return _columns_indices 742 743 with mrsm.Venv(get_connector_plugin(self.instance_connector)): 744 _columns_indices = ( 745 self.instance_connector.get_pipe_columns_indices(self, debug=debug) 746 if hasattr(self.instance_connector, 'get_pipe_columns_indices') 747 else None 748 ) 749 750 self._cache_value('_columns_indices', _columns_indices, debug=debug) 751 self._cache_value('_columns_indices_timestamp', now, debug=debug) 752 return {k: v for k, v in _columns_indices.items() if k and v} or {}
Return a dictionary mapping columns to index information.
1164def get_indices(self) -> Dict[str, str]: 1165 """ 1166 Return a dictionary mapping index keys to their names in the database. 1167 1168 Returns 1169 ------- 1170 A dictionary of index keys to index names. 1171 """ 1172 from meerschaum.connectors import get_connector_plugin 1173 with mrsm.Venv(get_connector_plugin(self.instance_connector)): 1174 if hasattr(self.instance_connector, 'get_pipe_index_names'): 1175 result = self.instance_connector.get_pipe_index_names(self) 1176 else: 1177 result = {} 1178 1179 return result
Return a dictionary mapping index keys to their names in the database.
Returns
- A dictionary of index keys to index names.
79def get_parameters( 80 self, 81 apply_symlinks: bool = True, 82 refresh: bool = False, 83 debug: bool = False, 84 _visited: 'Optional[set[mrsm.Pipe]]' = None, 85) -> Dict[str, Any]: 86 """ 87 Return the `parameters` dictionary of the pipe. 88 89 Parameters 90 ---------- 91 apply_symlinks: bool, default True 92 If `True`, resolve references to parameters from other pipes. 93 94 refresh: bool, default False 95 If `True`, pull the latest attributes for the pipe. 96 97 Returns 98 ------- 99 The pipe's parameters dictionary. 100 """ 101 from copy import deepcopy 102 from meerschaum.config._patch import apply_patch_to_config 103 from meerschaum.config._read_config import search_and_substitute_config 104 105 is_top_level = _visited is None 106 if _visited is None: 107 _visited = {self} 108 109 if refresh: 110 _ = self._invalidate_cache(hard=True) 111 with self._attributes_lock: 112 self._clear_cache_key('_attributes_sync_time', debug=debug) 113 self.__dict__.pop('_target', None) 114 ### Drop any memoized resolution so a later non-refresh call recomputes from fresh state. 115 _ = self.__dict__.pop('_resolved_parameters_raw', None) 116 _ = self.__dict__.pop('_resolved_parameters', None) 117 _ = self.__dict__.pop('_resolved_parameters_symlinks', None) 118 119 raw_parameters = self.attributes.get('parameters', {}) 120 if not apply_symlinks: 121 return raw_parameters 122 123 ### Resolving references + `{{ Pipe() }}` / `MRSM{}` symlinks is pure-Python but expensive 124 ### (it walks reference pipes and may build connectors), and `get_parameters` is a hot path 125 ### hit by `.dtypes`, `.columns`, `.precision`, etc. Memoize the resolved result, keyed on the 126 ### identity of the raw parameters dict: every mutation path (`update_parameters`, the setter, 127 ### `edit`) reassigns `_attributes['parameters']` to a new object, so identity changing is a 128 ### reliable invalidation signal. Schema is *not* part of this — dynamic-schema freshness is 129 ### handled separately by `get_columns_types`' TTL cache, so this is safe for dynamic pipes. 130 ### Only memoize the top-level entry (not nested reference resolution, which threads `_visited` 131 ### for cycle detection) and only the default symlink-resolving, non-refreshing call. 132 can_memoize = is_top_level and not refresh 133 if can_memoize and self.__dict__.get('_resolved_parameters_raw', None) is raw_parameters: 134 self._symlinks = self.__dict__.get('_resolved_parameters_symlinks', {}) 135 ### Return a copy so callers that mutate the result (e.g. `infer_dtypes(persist=True)`) 136 ### don't corrupt the memo. 137 return deepcopy(self.__dict__['_resolved_parameters']) 138 139 parameters = {} 140 for ref_pipe in self.references: 141 try: 142 if ref_pipe in _visited: 143 warn(f"Circular reference detected in {self}: chain involves {ref_pipe}.") 144 return search_and_substitute_config(raw_parameters) 145 146 _visited.add(ref_pipe) 147 if refresh: 148 _ = _cached_base_params.pop(ref_pipe, None) 149 base_params = _cached_base_params.get(ref_pipe, None) 150 if base_params is None: 151 base_params = ref_pipe.get_parameters( 152 apply_symlinks=apply_symlinks, 153 _visited=_visited, 154 debug=debug, 155 ) 156 _cached_base_params[ref_pipe] = base_params 157 if debug: 158 dprint(f"base_params from {ref_pipe} for {self}:") 159 mrsm.pprint(base_params) 160 else: 161 if debug: 162 dprint(f"Using cached base_params from {ref_pipe} for {self}") 163 except Exception as e: 164 warn(f"Failed to resolve reference pipe for {self}: {e}") 165 base_params = {} 166 167 parameters = apply_patch_to_config(parameters, base_params) 168 169 parameters = apply_patch_to_config(parameters, raw_parameters) 170 171 from meerschaum.utils.pipes import replace_pipes_syntax 172 self._symlinks = {} 173 174 def recursive_replace(obj: Any, path: tuple) -> Any: 175 if isinstance(obj, dict): 176 return {k: recursive_replace(v, path + (k,)) for k, v in obj.items()} 177 if isinstance(obj, list): 178 return [recursive_replace(elem, path + (i,)) for i, elem in enumerate(obj)] 179 if isinstance(obj, str): 180 substituted_val = replace_pipes_syntax(obj, _pipe=self) 181 if substituted_val != obj: 182 self._symlinks[path] = { 183 'original': obj, 184 'substituted': substituted_val, 185 } 186 return substituted_val 187 return obj 188 189 resolved_parameters = search_and_substitute_config(recursive_replace(parameters, tuple())) 190 191 if can_memoize: 192 ### Hold a reference to the raw dict so its identity can't be reused by a freed object, 193 ### and stash the symlinks captured above alongside the resolved result. 194 self.__dict__['_resolved_parameters_raw'] = raw_parameters 195 self.__dict__['_resolved_parameters'] = resolved_parameters 196 self.__dict__['_resolved_parameters_symlinks'] = self._symlinks 197 return deepcopy(resolved_parameters) 198 199 return resolved_parameters
Return the parameters dictionary of the pipe.
Parameters
- apply_symlinks (bool, default True):
If
True, resolve references to parameters from other pipes. - refresh (bool, default False):
If
True, pull the latest attributes for the pipe.
Returns
- The pipe's parameters dictionary.
352def get_dtypes( 353 self, 354 infer: bool = True, 355 refresh: bool = False, 356 debug: bool = False, 357) -> Dict[str, Any]: 358 """ 359 If defined, return the `dtypes` dictionary defined in `meerschaum.Pipe.parameters`. 360 361 Parameters 362 ---------- 363 infer: bool, default True 364 If `True`, include the implicit existing dtypes. 365 Else only return the explicitly configured dtypes (e.g. `Pipe.parameters['dtypes']`). 366 367 refresh: bool, default False 368 If `True`, invalidate any cache and return the latest known dtypes. 369 370 Returns 371 ------- 372 A dictionary mapping column names to dtypes. 373 """ 374 from meerschaum.config._patch import apply_patch_to_config 375 from meerschaum.utils.dtypes import MRSM_ALIAS_DTYPES 376 parameters = self.get_parameters(refresh=refresh, debug=debug) 377 configured_dtypes = parameters.get('dtypes', {}) 378 if debug: 379 dprint(f"Configured dtypes for {self}:") 380 mrsm.pprint(configured_dtypes) 381 382 remote_dtypes = ( 383 self.infer_dtypes(persist=False, refresh=refresh, debug=debug) 384 if infer 385 else {} 386 ) 387 if debug and infer: 388 dprint(f"Remote dtypes for {self}:") 389 mrsm.pprint(remote_dtypes) 390 391 patched_dtypes = apply_patch_to_config((remote_dtypes or {}), (configured_dtypes or {})) 392 393 dt_col = parameters.get('columns', {}).get('datetime', None) 394 primary_col = parameters.get('columns', {}).get('primary', None) 395 _dtypes = { 396 col: MRSM_ALIAS_DTYPES.get(typ, typ) 397 for col, typ in patched_dtypes.items() 398 if col and typ 399 } 400 if dt_col and dt_col not in configured_dtypes: 401 _dtypes[dt_col] = 'datetime' 402 if primary_col and parameters.get('autoincrement', False) and primary_col not in _dtypes: 403 _dtypes[primary_col] = 'int' 404 405 return _dtypes
If defined, return the dtypes dictionary defined in meerschaum.Pipe.parameters.
Parameters
- infer (bool, default True):
If
True, include the implicit existing dtypes. Else only return the explicitly configured dtypes (e.g.Pipe.parameters['dtypes']). - refresh (bool, default False):
If
True, invalidate any cache and return the latest known dtypes.
Returns
- A dictionary mapping column names to dtypes.
1182def update_parameters( 1183 self, 1184 parameters_patch: Dict[str, Any], 1185 persist: bool = True, 1186 debug: bool = False, 1187) -> mrsm.SuccessTuple: 1188 """ 1189 Apply a patch to a pipe's `parameters` dictionary. 1190 1191 Parameters 1192 ---------- 1193 parameters_patch: Dict[str, Any] 1194 The patch to be applied to `Pipe.parameters`. 1195 1196 persist: bool, default True 1197 If `True`, call `Pipe.edit()` to persist the new parameters. 1198 """ 1199 from meerschaum.config import apply_patch_to_config 1200 if 'parameters' not in self._attributes: 1201 self._attributes['parameters'] = {} 1202 1203 self._attributes['parameters'] = apply_patch_to_config( 1204 self._attributes['parameters'], 1205 parameters_patch, 1206 ) 1207 1208 if self.temporary: 1209 persist = False 1210 1211 if not persist: 1212 return True, "Success" 1213 1214 return self.edit(debug=debug)
Apply a patch to a pipe's parameters dictionary.
Parameters
- parameters_patch (Dict[str, Any]):
The patch to be applied to
Pipe.parameters. - persist (bool, default True):
If
True, callPipe.edit()to persist the new parameters.
412def validate_indices(self, debug: bool = False) -> mrsm.SuccessTuple: 413 """ 414 Return whether this pipe's index columns can actually be used as indices. 415 416 A `json` column holds `dict` or `list` values. Syncing sorts and deduplicates 417 on the index columns in Pandas (`Series.unique()`, `DataFrame.sort_values()`), 418 and both raise `TypeError: unhashable type: 'dict'` on container values. The 419 failure surfaces on the *second* sync — the first one has nothing to diff 420 against — so it is worth reporting up front instead of at that point. 421 422 Note this is a client-side limitation, not a database one: on PostgreSQL the 423 `json` dtype is stored as `JSONB`, which does support B-tree, hash, and GIN 424 indices. Non-unique performance indices declared under `indices` are pure 425 DDL and are not affected; only the index *roles* in `columns` are. 426 427 Returns 428 ------- 429 A `SuccessTuple` of whether the index columns are valid. 430 """ 431 columns = self.columns or {} 432 if not columns: 433 return True, "Success" 434 435 dtypes = self.get_dtypes(infer=False, debug=debug) 436 offenders = { 437 col_ix: col 438 for col_ix, col in columns.items() 439 if col and str(dtypes.get(col, '')).lower() in UNHASHABLE_DTYPES 440 } 441 if not offenders: 442 return True, "Success" 443 444 listing = ', '.join( 445 f"'{col}' (as the '{col_ix}' index)" 446 for col_ix, col in sorted(offenders.items()) 447 ) 448 return False, ( 449 f"Cannot use a 'json' column as an index column: {listing}.\n" 450 " JSON values are dictionaries or lists, which cannot be sorted or " 451 "deduplicated when syncing.\n" 452 " Either give the column a scalar dtype (e.g. 'str'), or move it out of " 453 "`columns` and\n" 454 " declare it under `indices` if you only need a database index on it." 455 )
Return whether this pipe's index columns can actually be used as indices.
A json column holds dict or list values. Syncing sorts and deduplicates
on the index columns in Pandas (Series.unique(), DataFrame.sort_values()),
and both raise TypeError: unhashable type: 'dict' on container values. The
failure surfaces on the second sync — the first one has nothing to diff
against — so it is worth reporting up front instead of at that point.
Note this is a client-side limitation, not a database one: on PostgreSQL the
json dtype is stored as JSONB, which does support B-tree, hash, and GIN
indices. Non-unique performance indices declared under indices are pure
DDL and are not affected; only the index roles in columns are.
Returns
- A
SuccessTupleof whether the index columns are valid.
755def get_id(self, **kw: Any) -> Union[int, str, None]: 756 """ 757 Fetch a pipe's ID from its instance connector. 758 If the pipe is not registered, return `None`. 759 """ 760 if self.temporary: 761 return None 762 763 from meerschaum.utils.venv import Venv 764 from meerschaum.connectors import get_connector_plugin 765 766 with Venv(get_connector_plugin(self.instance_connector)): 767 if hasattr(self.instance_connector, 'get_pipe_id'): 768 return self.instance_connector.get_pipe_id(self, **kw) 769 770 return None
Fetch a pipe's ID from its instance connector.
If the pipe is not registered, return None.
773@property 774def id(self) -> Union[int, str, uuid.UUID, None]: 775 """ 776 Fetch and cache a pipe's ID. 777 """ 778 _id = self._get_cached_value('_id', debug=self.debug) 779 if _id is None: 780 _id = self.get_id(debug=self.debug) 781 if _id is not None: 782 self._cache_value('_id', _id, debug=self.debug) 783 return _id
Fetch and cache a pipe's ID.
786def get_val_column(self, debug: bool = False) -> Union[str, None]: 787 """ 788 Return the name of the value column if it's defined, otherwise make an educated guess. 789 If not set in the `columns` dictionary, return the first numeric column that is not 790 an ID or datetime column. 791 If none may be found, return `None`. 792 793 Parameters 794 ---------- 795 debug: bool, default False: 796 Verbosity toggle. 797 798 Returns 799 ------- 800 Either a string or `None`. 801 """ 802 if debug: 803 dprint('Attempting to determine the value column...') 804 try: 805 val_name = self.get_columns('value') 806 except Exception: 807 val_name = None 808 if val_name is not None: 809 if debug: 810 dprint(f"Value column: {val_name}") 811 return val_name 812 813 cols = self.columns 814 if cols is None: 815 if debug: 816 dprint('No columns could be determined. Returning...') 817 return None 818 try: 819 dt_name = self.get_columns('datetime', error=False) 820 except Exception: 821 dt_name = None 822 try: 823 id_name = self.get_columns('id', errors=False) 824 except Exception: 825 id_name = None 826 827 if debug: 828 dprint(f"dt_name: {dt_name}") 829 dprint(f"id_name: {id_name}") 830 831 cols_types = self.get_columns_types(debug=debug) 832 if cols_types is None: 833 return None 834 if debug: 835 dprint(f"cols_types: {cols_types}") 836 if dt_name is not None: 837 cols_types.pop(dt_name, None) 838 if id_name is not None: 839 cols_types.pop(id_name, None) 840 841 candidates = [] 842 candidate_keywords = {'float', 'double', 'precision', 'int', 'numeric',} 843 for search_term in candidate_keywords: 844 for col, typ in cols_types.items(): 845 if search_term in typ.lower(): 846 candidates.append(col) 847 break 848 if not candidates: 849 if debug: 850 dprint("No value column could be determined.") 851 return None 852 853 return candidates[0]
Return the name of the value column if it's defined, otherwise make an educated guess.
If not set in the columns dictionary, return the first numeric column that is not
an ID or datetime column.
If none may be found, return None.
Parameters
- debug (bool, default False:): Verbosity toggle.
Returns
- Either a string or
None.
856@property 857def parents(self) -> List[mrsm.Pipe]: 858 """ 859 Return a list of `meerschaum.Pipe` objects to be designated as parents. 860 """ 861 _cached_parents = self.__dict__.get('_parents', None) 862 if _cached_parents is not None: 863 return _cached_parents 864 865 from meerschaum.utils.pipes import get_pipe_from_value 866 base_params = self.get_parameters() 867 key = 'parents' if 'parents' in base_params else 'parent' 868 parents_refs = base_params.get(key, None) or [] 869 if isinstance(parents_refs, str) or isinstance(parents_refs, dict): 870 parents_refs = [parents_refs] 871 872 if not parents_refs: 873 return [] 874 875 self._parents = [get_pipe_from_value(val, _pipe=self) for val in parents_refs] 876 return self._parents
Return a list of meerschaum.Pipe objects to be designated as parents.
879@property 880def parent(self) -> Union[mrsm.Pipe, None]: 881 """ 882 Return the first pipe in `self.parents` or `None`. 883 """ 884 _parents = self.parents 885 if not _parents: 886 return None 887 888 return _parents[0]
Return the first pipe in self.parents or None.
924@property 925def children(self) -> List[mrsm.Pipe]: 926 """ 927 Return a list of `meerschaum.Pipe` objects to be designated as children. 928 """ 929 _cached_children = self.__dict__.get('_children', None) 930 if _cached_children is not None: 931 return _cached_children 932 933 from meerschaum.utils.pipes import get_pipe_from_value 934 base_params = self.get_parameters() 935 key = 'children' if 'children' in base_params else 'child' 936 children_refs = base_params.get(key, None) or [] 937 if isinstance(children_refs, str) or isinstance(children_refs, dict): 938 children_refs = [children_refs] 939 940 if not children_refs: 941 return [] 942 943 self._children = [get_pipe_from_value(val, _pipe=self) for val in children_refs] 944 return self._children
Return a list of meerschaum.Pipe objects to be designated as children.
947@property 948def child(self) -> mrsm.Pipe | None: 949 """ 950 Return the first pipe in `self.children` or None. 951 """ 952 _children = self.children 953 if not _children: 954 return None 955 956 return _children[0]
Return the first pipe in self.children or None.
1016@property 1017def reference(self) -> mrsm.Pipe | None: 1018 """ 1019 Return the first pipe in `self.references` or None. 1020 """ 1021 _references = self.references 1022 if not _references: 1023 return None 1024 1025 return _references[0]
Return the first pipe in self.references or None.
993@property 994def references(self) -> List[mrsm.Pipe]: 995 """ 996 Return a list of `meerschaum.Pipe` objects to be designated as references. 997 """ 998 _cached_references = self.__dict__.get('_references', None) 999 if _cached_references is not None: 1000 return _cached_references 1001 1002 from meerschaum.utils.pipes import get_pipe_from_value 1003 base_params = self.get_parameters(apply_symlinks=False) 1004 key = 'references' if 'references' in base_params else 'reference' 1005 refs = base_params.get(key, None) or [] 1006 if isinstance(refs, str) or isinstance(refs, dict): 1007 refs = [refs] 1008 1009 if not refs: 1010 return [] 1011 1012 self._references = [get_pipe_from_value(val, _pipe=self) for val in refs] 1013 return self._references
Return a list of meerschaum.Pipe objects to be designated as references.
1063@property 1064def target(self) -> str: 1065 """ 1066 The target table name. 1067 Registered pipes do not memoize guessed defaults while metadata failures are outstanding. 1068 You can set the target name under on of the following keys 1069 (checked in this order): 1070 - `target` 1071 - `target_name` 1072 - `target_table` 1073 - `target_table_name` 1074 """ 1075 cached_target = self.__dict__.get('_target', None) 1076 if cached_target: 1077 return cached_target 1078 1079 load_failures = self.__dict__.get('_attributes_load_failures', 0) 1080 params = self.parameters 1081 target_val = params.get('target', None) 1082 if target_val: 1083 self.__dict__['_target'] = target_val 1084 return target_val 1085 1086 default_target = self._target_legacy() 1087 default_targets = {default_target} 1088 potential_keys = ('target_name', 'target_table', 'target_table_name') 1089 _target = None 1090 for k in potential_keys: 1091 if k in params: 1092 _target = params[k] 1093 break 1094 1095 _target = _target or default_target 1096 1097 if self.instance_connector.type == 'sql': 1098 from meerschaum.utils.sql import truncate_item_name 1099 truncated_target = truncate_item_name(_target, self.instance_connector.flavor) 1100 default_targets.add(truncated_target) 1101 warned_target = self.__dict__.get('_warned_target', False) 1102 if truncated_target != _target and not warned_target: 1103 if self.instance_connector.flavor not in ('oracle', 'mysql', 'mariadb'): 1104 warn( 1105 f"The target '{_target}' is too long for '{self.instance_connector.flavor}', " 1106 + f"will use {truncated_target} instead." 1107 ) 1108 self.__dict__['_warned_target'] = True 1109 _target = truncated_target 1110 1111 if _target not in default_targets: 1112 self.target = _target 1113 1114 # Parameter resolution may retry a failed load; its first snapshot is still stale. 1115 resolved_cleanly = load_failures == self.__dict__.get('_attributes_load_failures', 0) 1116 if (params and resolved_cleanly) or self.temporary or self.id is None: 1117 self.__dict__['_target'] = _target 1118 return _target
The target table name. Registered pipes do not memoize guessed defaults while metadata failures are outstanding. You can set the target name under on of the following keys (checked in this order):
targettarget_nametarget_tabletarget_table_name
1142def guess_datetime(self) -> Union[str, None]: 1143 """ 1144 Try to determine a pipe's datetime column. 1145 """ 1146 _dtypes = self.dtypes 1147 1148 ### Abort if the user explictly disallows a datetime index. 1149 if 'datetime' in _dtypes: 1150 if _dtypes['datetime'] is None: 1151 return None 1152 1153 from meerschaum.utils.dtypes import are_dtypes_equal 1154 dt_cols = [ 1155 col 1156 for col, typ in _dtypes.items() 1157 if are_dtypes_equal(typ, 'datetime') 1158 ] 1159 if not dt_cols: 1160 return None 1161 return dt_cols[0]
Try to determine a pipe's datetime column.
1306@property 1307def precision(self) -> Dict[str, Union[str, int]]: 1308 """ 1309 Return the configured or detected precision. 1310 """ 1311 return self.get_precision(debug=self.debug)
Return the configured or detected precision.
1217def get_precision(self, debug: bool = False) -> Dict[str, Union[str, int]]: 1218 """ 1219 Return the timestamp precision unit and interval for the `datetime` axis. 1220 """ 1221 from meerschaum.utils.dtypes import ( 1222 MRSM_PRECISION_UNITS_SCALARS, 1223 MRSM_PRECISION_UNITS_ALIASES, 1224 MRSM_PD_DTYPES, 1225 are_dtypes_equal, 1226 ) 1227 from meerschaum._internal.static import STATIC_CONFIG 1228 1229 _precision = self._get_cached_value('precision', debug=debug) 1230 if _precision: 1231 if debug: 1232 dprint(f"Returning cached precision: {_precision}") 1233 return _precision 1234 1235 parameters = self.parameters 1236 _precision = parameters.get('precision', {}) 1237 if isinstance(_precision, str): 1238 _precision = {'unit': _precision} 1239 default_precision_unit = STATIC_CONFIG['dtypes']['datetime']['default_precision_unit'] 1240 1241 if not _precision: 1242 1243 dt_col = parameters.get('columns', {}).get('datetime', None) 1244 if not dt_col and self.autotime: 1245 dt_col = mrsm.get_config('pipes', 'autotime', 'column_name_if_datetime_missing') 1246 if not dt_col: 1247 if debug: 1248 dprint(f"No datetime axis, returning default precision '{default_precision_unit}'.") 1249 return {'unit': default_precision_unit} 1250 1251 dt_typ = self.dtypes.get(dt_col, 'datetime') 1252 if are_dtypes_equal(dt_typ, 'datetime'): 1253 if dt_typ == 'datetime': 1254 dt_typ = MRSM_PD_DTYPES['datetime'] 1255 if debug: 1256 dprint(f"Datetime type is `datetime`, assuming {dt_typ} precision.") 1257 1258 _precision = { 1259 'unit': ( 1260 dt_typ 1261 .split('[', maxsplit=1)[-1] 1262 .split(',', maxsplit=1)[0] 1263 .split(' ', maxsplit=1)[0] 1264 ).rstrip(']') 1265 } 1266 1267 if debug: 1268 dprint(f"Extracted precision '{_precision['unit']}' from type '{dt_typ}'.") 1269 1270 elif are_dtypes_equal(dt_typ, 'int'): 1271 _precision = { 1272 'unit': ( 1273 'second' 1274 if '32' in dt_typ 1275 else default_precision_unit 1276 ) 1277 } 1278 elif are_dtypes_equal(dt_typ, 'date'): 1279 if debug: 1280 dprint("Datetime axis is 'date', falling back to 'day' precision.") 1281 _precision = {'unit': 'day'} 1282 1283 precision_unit = _precision.get('unit', default_precision_unit) 1284 precision_interval = _precision.get('interval', None) 1285 true_precision_unit = MRSM_PRECISION_UNITS_ALIASES.get(precision_unit, precision_unit) 1286 if true_precision_unit is None: 1287 if debug: 1288 dprint(f"No precision could be determined, falling back to '{default_precision_unit}'.") 1289 true_precision_unit = default_precision_unit 1290 1291 if true_precision_unit not in MRSM_PRECISION_UNITS_SCALARS: 1292 from meerschaum.utils.misc import items_str 1293 raise ValueError( 1294 f"Invalid precision unit '{true_precision_unit}'.\n" 1295 "Accepted values are " 1296 f"{items_str(list(MRSM_PRECISION_UNITS_SCALARS) + list(MRSM_PRECISION_UNITS_ALIASES))}." 1297 ) 1298 1299 _precision = {'unit': true_precision_unit} 1300 if precision_interval: 1301 _precision['interval'] = precision_interval 1302 self._cache_value('precision', _precision, debug=debug) 1303 return self._precision
Return the timestamp precision unit and interval for the datetime axis.
12def show( 13 self, 14 nopretty: bool = False, 15 debug: bool = False, 16 **kw 17) -> SuccessTuple: 18 """ 19 Show attributes of a Pipe. 20 21 Parameters 22 ---------- 23 nopretty: bool, default False 24 If `True`, simply print the JSON of the pipe's attributes. 25 26 debug: bool, default False 27 Verbosity toggle. 28 29 Returns 30 ------- 31 A `SuccessTuple` of success, message. 32 33 """ 34 import json 35 from meerschaum.utils.formatting import ( 36 pprint, make_header, ANSI, highlight_pipes, fill_ansi, get_console, 37 ) 38 from meerschaum.utils.packages import import_rich, attempt_import 39 from meerschaum.utils.warnings import info 40 attributes_json = json.dumps(self.attributes) 41 if not nopretty: 42 _to_print = f"Attributes for {self}:" 43 if ANSI: 44 _to_print = fill_ansi(highlight_pipes(make_header(_to_print)), 'magenta') 45 print(_to_print) 46 rich = import_rich() 47 rich_json = attempt_import('rich.json') 48 get_console().print(rich_json.JSON(attributes_json)) 49 else: 50 print(_to_print) 51 else: 52 print(attributes_json) 53 54 return True, "Success"
Show attributes of a Pipe.
Parameters
- nopretty (bool, default False):
If
True, simply print the JSON of the pipe's attributes. - debug (bool, default False): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
21def edit( 22 self, 23 patch: bool = False, 24 interactive: bool = False, 25 debug: bool = False, 26 **kw: Any 27) -> SuccessTuple: 28 """ 29 Edit a Pipe's configuration. 30 31 Parameters 32 ---------- 33 patch: bool, default False 34 If `patch` is True, update parameters by cascading rather than overwriting. 35 interactive: bool, default False 36 If `True`, open an editor for the user to make changes to the pipe's YAML file. 37 debug: bool, default False 38 Verbosity toggle. 39 40 Returns 41 ------- 42 A `SuccessTuple` of success, message. 43 44 """ 45 from meerschaum.utils.venv import Venv 46 from meerschaum.connectors import get_connector_plugin 47 48 if self.temporary: 49 return False, "Cannot edit pipes created with `temporary=True` (read-only)." 50 51 self._invalidate_cache(hard=True, debug=debug) 52 53 if hasattr(self, '_symlinks'): 54 from meerschaum.utils.misc import get_val_from_dict_path, set_val_in_dict_path 55 for path, vals in self._symlinks.items(): 56 current_val = get_val_from_dict_path(self.parameters, path) 57 if current_val == vals['substituted']: 58 set_val_in_dict_path(self.parameters, path, vals['original']) 59 60 if not interactive: 61 ### Refuse an index column which cannot be sorted or deduplicated. 62 indices_success, indices_msg = self.validate_indices(debug=debug) 63 if not indices_success: 64 return indices_success, indices_msg 65 66 with Venv(get_connector_plugin(self.instance_connector)): 67 return self.instance_connector.edit_pipe(self, patch=patch, debug=debug, **kw) 68 69 import meerschaum.config.paths as paths 70 from meerschaum.utils.misc import edit_file 71 parameters_filename = str(self) + '.yaml' 72 parameters_path = paths.PIPES_CACHE_RESOURCES_PATH / parameters_filename 73 74 from meerschaum.utils.yaml import yaml 75 76 edit_text = f"Edit the parameters for {self}" 77 edit_top = '#' * (len(edit_text) + 4) 78 edit_header = edit_top + f'\n# {edit_text} #\n' + edit_top + '\n\n' 79 80 from meerschaum.config import get_config 81 parameters = dict(get_config('pipes', 'parameters', patch=True)) 82 from meerschaum.config._patch import apply_patch_to_config 83 raw_parameters = self.attributes.get('parameters', {}) 84 parameters = apply_patch_to_config(parameters, raw_parameters) 85 86 ### write parameters to yaml file 87 with open(parameters_path, 'w+') as f: 88 f.write(edit_header) 89 yaml.dump(parameters, stream=f, sort_keys=False) 90 91 ### only quit editing if yaml is valid 92 editing = True 93 while editing: 94 edit_file(parameters_path) 95 try: 96 with open(parameters_path, 'r') as f: 97 file_parameters = yaml.load(f.read()) 98 except Exception as e: 99 from meerschaum.utils.warnings import warn 100 warn(f"Invalid format defined for '{self}':\n\n{e}") 101 input(f"Press [Enter] to correct the configuration for '{self}': ") 102 else: 103 editing = False 104 105 self.parameters = file_parameters 106 107 if debug: 108 from meerschaum.utils.formatting import pprint 109 pprint(self.parameters) 110 111 with Venv(get_connector_plugin(self.instance_connector)): 112 return self.instance_connector.edit_pipe(self, patch=patch, debug=debug, **kw)
Edit a Pipe's configuration.
Parameters
- patch (bool, default False):
If
patchis True, update parameters by cascading rather than overwriting. - interactive (bool, default False):
If
True, open an editor for the user to make changes to the pipe's YAML file. - debug (bool, default False): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
115def edit_definition( 116 self, 117 yes: bool = False, 118 noask: bool = False, 119 force: bool = False, 120 debug : bool = False, 121 **kw : Any 122) -> SuccessTuple: 123 """ 124 Edit a pipe's definition file and update its configuration. 125 **NOTE:** This function is interactive and should not be used in automated scripts! 126 127 Returns 128 ------- 129 A `SuccessTuple` of success, message. 130 131 """ 132 if self.temporary: 133 return False, "Cannot edit pipes created with `temporary=True` (read-only)." 134 135 from meerschaum.connectors import instance_types 136 if (self.connector is None or isinstance(self.connector, str)) or self.connector.type not in instance_types: 137 return self.edit(interactive=True, debug=debug, **kw) 138 139 import json 140 from meerschaum.utils.warnings import info, warn 141 from meerschaum.utils.debug import dprint 142 from meerschaum.config._patch import apply_patch_to_config 143 from meerschaum.utils.misc import edit_file 144 145 _parameters = self.parameters 146 if 'fetch' not in _parameters: 147 _parameters['fetch'] = {} 148 149 def _edit_api(): 150 from meerschaum.utils.prompt import prompt, yes_no 151 info( 152 f"Please enter the keys of the source pipe from '{self.connector}'.\n" + 153 "Type 'None' for None, or empty when there is no default. Press [CTRL+C] to skip." 154 ) 155 156 _keys = { 'connector_keys' : None, 'metric_key' : None, 'location_key' : None } 157 for k in _keys: 158 _keys[k] = _parameters['fetch'].get(k, None) 159 160 for k, v in _keys.items(): 161 try: 162 _keys[k] = prompt(k.capitalize().replace('_', ' ') + ':', icon=True, default=v) 163 except KeyboardInterrupt: 164 continue 165 if _keys[k] in ('', 'None', '\'None\'', '[None]'): 166 _keys[k] = None 167 168 _parameters['fetch'] = apply_patch_to_config(_parameters['fetch'], _keys) 169 170 info("You may optionally specify additional filter parameters as JSON.") 171 print(" Parameters are translated into a 'WHERE x AND y' clause, and lists are IN clauses.") 172 print(" For example, the following JSON would correspond to 'WHERE x = 1 AND y IN (2, 3)':") 173 print(json.dumps({'x': 1, 'y': [2, 3]}, indent=2, separators=(',', ': '))) 174 if force or yes_no( 175 "Would you like to add additional filter parameters?", 176 yes=yes, noask=noask 177 ): 178 import meerschaum.config.paths as paths 179 definition_filename = str(self) + '.json' 180 definition_path = paths.PIPES_CACHE_RESOURCES_PATH / definition_filename 181 try: 182 definition_path.touch() 183 with open(definition_path, 'w+') as f: 184 json.dump(_parameters.get('fetch', {}).get('params', {}), f, indent=2) 185 except Exception as e: 186 return False, f"Failed writing file '{definition_path}':\n" + str(e) 187 188 _params = None 189 while True: 190 edit_file(definition_path) 191 try: 192 with open(definition_path, 'r') as f: 193 _params = json.load(f) 194 except Exception as e: 195 warn(f'Failed to read parameters JSON:\n{e}', stack=False) 196 if force or yes_no( 197 "Would you like to try again?\n " 198 + "If not, the parameters JSON file will be ignored.", 199 noask=noask, yes=yes 200 ): 201 continue 202 _params = None 203 break 204 if _params is not None: 205 if 'fetch' not in _parameters: 206 _parameters['fetch'] = {} 207 _parameters['fetch']['params'] = _params 208 209 self.parameters = _parameters 210 return True, "Success" 211 212 def _edit_sql(): 213 import textwrap 214 import meerschaum.config.paths as paths 215 from meerschaum.utils.misc import edit_file 216 definition_filename = str(self) + '.sql' 217 definition_path = paths.PIPES_CACHE_RESOURCES_PATH / definition_filename 218 219 sql_definition = _parameters['fetch'].get('definition', None) 220 if sql_definition is None: 221 sql_definition = '' 222 sql_definition = textwrap.dedent(sql_definition).lstrip() 223 224 try: 225 definition_path.touch() 226 with open(definition_path, 'w+') as f: 227 f.write(sql_definition) 228 except Exception as e: 229 return False, f"Failed writing file '{definition_path}':\n" + str(e) 230 231 edit_file(definition_path) 232 try: 233 with open(definition_path, 'r', encoding='utf-8') as f: 234 file_definition = f.read() 235 except Exception as e: 236 return False, f"Failed reading file '{definition_path}':\n" + str(e) 237 238 if sql_definition == file_definition: 239 return False, f"No changes made to definition for {self}." 240 241 if ' ' not in file_definition: 242 return False, f"Invalid SQL definition for {self}." 243 244 if debug: 245 dprint("Read SQL definition:\n\n" + file_definition) 246 _parameters['fetch']['definition'] = file_definition 247 self.parameters = _parameters 248 return True, "Success" 249 250 locals()['_edit_' + str(self.connector.type)]() 251 return self.edit(interactive=False, debug=debug, **kw)
Edit a pipe's definition file and update its configuration. NOTE: This function is interactive and should not be used in automated scripts!
Returns
- A
SuccessTupleof success, message.
13def update(self, *args, **kw) -> SuccessTuple: 14 """ 15 Update a pipe's parameters in its instance. 16 """ 17 kw['interactive'] = False 18 return self.edit(*args, **kw)
Update a pipe's parameters in its instance.
41def sync( 42 self, 43 df: Union[ 44 pd.DataFrame, 45 Dict[str, List[Any]], 46 List[Dict[str, Any]], 47 str, 48 InferFetch 49 ] = InferFetch, 50 begin: Union[datetime, int, str, None] = '', 51 end: Union[datetime, int, None] = None, 52 force: bool = False, 53 retries: int = 10, 54 min_seconds: int = 1, 55 check_existing: bool = True, 56 enforce_dtypes: bool = True, 57 blocking: bool = True, 58 workers: Optional[int] = None, 59 callback: Optional[Callable[[Tuple[bool, str]], Any]] = None, 60 error_callback: Optional[Callable[[Exception], Any]] = None, 61 chunksize: Optional[int] = -1, 62 sync_chunks: bool = True, 63 debug: bool = False, 64 _inplace: bool = True, 65 **kw: Any 66) -> SuccessTuple: 67 """ 68 Fetch new data from the source and update the pipe's table with new data. 69 70 Get new remote data via fetch, get existing data in the same time period, 71 and merge the two, only keeping the unseen data. 72 73 Parameters 74 ---------- 75 df: Union[None, pd.DataFrame, Dict[str, List[Any]]], default None 76 An optional DataFrame to sync into the pipe. Defaults to `None`. 77 If `df` is a string, it will be parsed via `meerschaum.utils.dataframe.parse_simple_lines()`. 78 79 begin: Union[datetime, int, str, None], default '' 80 Optionally specify the earliest datetime to search for data. 81 82 end: Union[datetime, int, str, None], default None 83 Optionally specify the latest datetime to search for data. 84 85 force: bool, default False 86 If `True`, keep trying to sync untul `retries` attempts. 87 88 retries: int, default 10 89 If `force`, how many attempts to try syncing before declaring failure. 90 91 min_seconds: Union[int, float], default 1 92 If `force`, how many seconds to sleep between retries. Defaults to `1`. 93 94 check_existing: bool, default True 95 If `True`, pull and diff with existing data from the pipe. 96 97 enforce_dtypes: bool, default True 98 If `True`, enforce dtypes on incoming data. 99 Set this to `False` if the incoming rows are expected to be of the correct dtypes. 100 101 blocking: bool, default True 102 If `True`, wait for sync to finish and return its result, otherwise 103 asyncronously sync (oxymoron?) and return success. Defaults to `True`. 104 Only intended for specific scenarios. 105 106 workers: Optional[int], default None 107 If provided and the instance connector is thread-safe 108 (`pipe.instance_connector.IS_THREAD_SAFE is True`), 109 limit concurrent sync to this many threads. 110 111 callback: Optional[Callable[[Tuple[bool, str]], Any]], default None 112 Callback function which expects a SuccessTuple as input. 113 Only applies when `blocking=False`. 114 115 error_callback: Optional[Callable[[Exception], Any]], default None 116 Callback function which expects an Exception as input. 117 Only applies when `blocking=False`. 118 119 chunksize: int, default -1 120 Specify the number of rows to sync per chunk. 121 If `-1`, resort to system configuration (default is `900`). 122 A `chunksize` of `None` will sync all rows in one transaction. 123 124 sync_chunks: bool, default True 125 If possible, sync chunks while fetching them into memory. 126 127 debug: bool, default False 128 Verbosity toggle. Defaults to False. 129 130 Returns 131 ------- 132 A `SuccessTuple` of success (`bool`) and message (`str`). 133 """ 134 from meerschaum.utils.debug import dprint, _checkpoint 135 from meerschaum.utils.formatting import get_console 136 from meerschaum.utils.venv import Venv 137 from meerschaum.connectors import get_connector_plugin 138 from meerschaum.utils.misc import df_is_chunk_generator, filter_keywords, filter_arguments 139 from meerschaum.utils.pool import get_pool 140 from meerschaum.config import get_config 141 from meerschaum.utils.dtypes import are_dtypes_equal, get_current_timestamp 142 143 if (callback is not None or error_callback is not None) and blocking: 144 warn("Callback functions are only executed when blocking = False. Ignoring...") 145 146 _checkpoint(_total=2, **kw) 147 148 if chunksize == 0: 149 chunksize = None 150 sync_chunks = False 151 152 begin, end = self.parse_date_bounds(begin, end) 153 kw.update({ 154 'begin': begin, 155 'end': end, 156 'force': force, 157 'retries': retries, 158 'min_seconds': min_seconds, 159 'check_existing': check_existing, 160 'blocking': blocking, 161 'workers': workers, 162 'callback': callback, 163 'error_callback': error_callback, 164 'sync_chunks': sync_chunks, 165 'chunksize': chunksize, 166 'safe_copy': True, 167 }) 168 169 self._invalidate_cache(debug=debug) 170 self._cache_value('sync_ts', get_current_timestamp('ms'), debug=debug) 171 172 def _sync( 173 p: mrsm.Pipe, 174 df: Union[ 175 'pd.DataFrame', 176 Dict[str, List[Any]], 177 List[Dict[str, Any]], 178 str, 179 InferFetch 180 ] = InferFetch, 181 ) -> SuccessTuple: 182 if df is None: 183 p._invalidate_cache(debug=debug) 184 return ( 185 False, 186 f"You passed `None` instead of data into `sync()` for {p}.\n" 187 + "Omit the DataFrame to infer fetching.", 188 ) 189 ### Ensure that Pipe is registered. 190 if not p.temporary and p.id is None: 191 ### NOTE: This may trigger an interactive session for plugins! 192 register_success, register_msg = p.register(debug=debug) 193 if not register_success: 194 if 'already' not in register_msg: 195 p._invalidate_cache(debug=debug) 196 return register_success, register_msg 197 198 ### Checked here as well as in `register()` so that a pipe registered before 199 ### this validation existed reports the reason rather than failing deep in 200 ### Pandas with `unhashable type: 'dict'`. 201 indices_success, indices_msg = p.validate_indices(debug=debug) 202 if not indices_success: 203 p._invalidate_cache(debug=debug) 204 return indices_success, indices_msg 205 206 if isinstance(df, str): 207 from meerschaum.utils.dataframe import parse_simple_lines 208 df = parse_simple_lines(df) 209 210 ### If connector is a plugin with a `sync()` method, return that instead. 211 ### If the plugin does not have a `sync()` method but does have a `fetch()` method, 212 ### use that instead. 213 ### NOTE: The DataFrame must be omitted for the plugin sync method to apply. 214 ### If a DataFrame is provided, continue as expected. 215 if hasattr(df, 'MRSM_INFER_FETCH'): 216 try: 217 if isinstance(p.connector, str): 218 if ':' not in p.connector_keys: 219 return True, f"{p} does not support fetching; nothing to do." 220 221 msg = f"{p} does not have a valid connector." 222 if p.connector_keys.startswith('plugin:'): 223 msg += f"\n Perhaps {p.connector_keys} has a syntax error?" 224 p._invalidate_cache(debug=debug) 225 return False, msg 226 except Exception: 227 p._invalidate_cache(debug=debug) 228 return False, f"Unable to create the connector for {p}." 229 230 ### Sync in place if possible. 231 if ( 232 str(self.connector) == str(self.instance_connector) 233 and 234 hasattr(self.instance_connector, 'sync_pipe_inplace') 235 and 236 _inplace 237 and 238 get_config('system', 'experimental', 'inplace_sync') 239 ): 240 with Venv(get_connector_plugin(self.instance_connector)): 241 p._invalidate_cache(debug=debug) 242 _args, _kwargs = filter_arguments( 243 p.instance_connector.sync_pipe_inplace, 244 p, 245 debug=debug, 246 **kw 247 ) 248 return self.instance_connector.sync_pipe_inplace( 249 *_args, 250 **_kwargs 251 ) 252 253 ### Activate and invoke `sync(pipe)` for plugin connectors with `sync` methods. 254 try: 255 if getattr(p.connector, 'sync', None) is not None: 256 with Venv(get_connector_plugin(p.connector), debug=debug): 257 _args, _kwargs = filter_arguments( 258 p.connector.sync, 259 p, 260 debug=debug, 261 **kw 262 ) 263 return_tuple = p.connector.sync(*_args, **_kwargs) 264 p._invalidate_cache(debug=debug) 265 if not isinstance(return_tuple, tuple): 266 return_tuple = ( 267 False, 268 f"Plugin '{p.connector.label}' returned non-tuple value: {return_tuple}" 269 ) 270 return return_tuple 271 272 except Exception as e: 273 get_console().print_exception() 274 msg = f"Failed to sync {p} with exception: '" + str(e) + "'" 275 if debug: 276 error(msg, silent=False) 277 p._invalidate_cache(debug=debug) 278 return False, msg 279 280 ### Fetch the dataframe from the connector's `fetch()` method. 281 try: 282 with Venv(get_connector_plugin(p.connector), debug=debug): 283 df = p.fetch( 284 **filter_keywords( 285 p.fetch, 286 debug=debug, 287 **kw 288 ) 289 ) 290 kw['safe_copy'] = False 291 except Exception as e: 292 get_console().print_exception( 293 suppress=[ 294 'meerschaum/core/Pipe/_sync.py', 295 'meerschaum/core/Pipe/_fetch.py', 296 ] 297 ) 298 msg = f"Failed to fetch data from {p.connector}:\n {e}" 299 df = None 300 301 if df is None: 302 p._invalidate_cache(debug=debug) 303 return False, f"No data were fetched for {p}." 304 305 if isinstance(df, list): 306 if len(df) == 0: 307 return True, f"No new rows were returned for {p}." 308 309 ### May be a chunk hook results list. 310 if isinstance(df[0], tuple): 311 success = all([_success for _success, _ in df]) 312 message = '\n'.join([_message for _, _message in df]) 313 return success, message 314 315 if df is True: 316 p._invalidate_cache(debug=debug) 317 return True, f"{p} is being synced in parallel." 318 319 ### CHECKPOINT: Retrieved the DataFrame. 320 _checkpoint(**kw) 321 322 ### Allow for dataframe generators or iterables. 323 if df_is_chunk_generator(df): 324 kw['workers'] = p.get_num_workers(kw.get('workers', None)) 325 dt_col = p.columns.get('datetime', None) 326 pool = get_pool(workers=kw.get('workers', 1)) 327 if debug: 328 dprint(f"Received {type(df)}. Attempting to sync first chunk...") 329 330 try: 331 chunk = next(df) 332 except StopIteration: 333 return True, "Received an empty generator; nothing to do." 334 335 chunk_success, chunk_msg = _sync(p, chunk) 336 chunk_msg = '\n' + self._get_chunk_label(chunk, dt_col) + '\n' + chunk_msg 337 if not chunk_success: 338 return chunk_success, f"Unable to sync initial chunk for {p}:\n{chunk_msg}" 339 if debug: 340 dprint("Successfully synced the first chunk, attemping the rest...") 341 342 def _process_chunk(_chunk): 343 _chunk_attempts = 0 344 _max_chunk_attempts = 3 345 while _chunk_attempts < _max_chunk_attempts: 346 try: 347 _chunk_success, _chunk_msg = _sync(p, _chunk) 348 except Exception as e: 349 _chunk_success, _chunk_msg = False, str(e) 350 if _chunk_success: 351 break 352 _chunk_attempts += 1 353 _sleep_seconds = _chunk_attempts ** 2 354 warn( 355 ( 356 f"Failed to sync chunk to {self} " 357 + f"(attempt {_chunk_attempts} / {_max_chunk_attempts}).\n" 358 + f"Sleeping for {_sleep_seconds} second" 359 + ('s' if _sleep_seconds != 1 else '') 360 + f":\n{_chunk_msg}" 361 ), 362 stack=False, 363 ) 364 time.sleep(_sleep_seconds) 365 366 num_rows_str = ( 367 f"{num_rows:,} rows" 368 if (num_rows := len(_chunk)) != 1 369 else f"{num_rows} row" 370 ) 371 _chunk_msg = ( 372 ( 373 "Synced" 374 if _chunk_success 375 else "Failed to sync" 376 ) + f" a chunk ({num_rows_str}) to {p}:\n" 377 + self._get_chunk_label(_chunk, dt_col) 378 + '\n' 379 + _chunk_msg 380 ) 381 382 mrsm.pprint((_chunk_success, _chunk_msg), calm=True) 383 return _chunk_success, _chunk_msg 384 385 results = sorted( 386 [(chunk_success, chunk_msg)] + ( 387 list(pool.imap(_process_chunk, df)) 388 if ( 389 not df_is_chunk_generator(chunk) # Handle nested generators. 390 and kw.get('workers', 1) != 1 391 ) 392 else list( 393 _process_chunk(_child_chunks) 394 for _child_chunks in df 395 ) 396 ) 397 ) 398 chunk_messages = [chunk_msg for _, chunk_msg in results] 399 success_bools = [chunk_success for chunk_success, _ in results] 400 num_successes = len([chunk_success for chunk_success, _ in results if chunk_success]) 401 num_failures = len([chunk_success for chunk_success, _ in results if not chunk_success]) 402 success = all(success_bools) 403 msg = ( 404 'Synced ' 405 + f'{len(chunk_messages):,} chunk' 406 + ('s' if len(chunk_messages) != 1 else '') 407 + f' to {p}\n({num_successes} succeeded, {num_failures} failed):\n\n' 408 + '\n\n'.join(chunk_messages).lstrip().rstrip() 409 ).lstrip().rstrip() 410 return success, msg 411 412 ### Cast to a dataframe and ensure datatypes are what we expect. 413 dtypes = p.get_dtypes(debug=debug) 414 df = p.enforce_dtypes( 415 df, 416 chunksize=chunksize, 417 enforce=enforce_dtypes, 418 dtypes=dtypes, 419 debug=debug, 420 ) 421 if p.autotime: 422 dt_col = p.columns.get('datetime', None) 423 ts_col = dt_col or mrsm.get_config( 424 'pipes', 'autotime', 'column_name_if_datetime_missing' 425 ) 426 ts_typ = dtypes.get(ts_col, 'datetime') if ts_col else 'datetime' 427 if ts_col and hasattr(df, 'columns') and ts_col not in df.columns: 428 precision = p.get_precision(debug=debug) 429 now = get_current_timestamp( 430 precision_unit=precision.get( 431 'unit', 432 STATIC_CONFIG['dtypes']['datetime']['default_precision_unit'] 433 ), 434 precision_interval=precision.get('interval', 1), 435 round_to=(precision.get('round_to', 'down')), 436 as_int=(are_dtypes_equal(ts_typ, 'int')), 437 ) 438 if debug: 439 dprint(f"Adding current timestamp to dataframe synced to {p}: {now}") 440 441 df[ts_col] = now 442 kw['check_existing'] = dt_col is not None 443 444 ### Capture special columns. 445 capture_success, capture_msg = self._persist_new_special_columns( 446 df, 447 dtypes=dtypes, 448 debug=debug, 449 ) 450 if not capture_success: 451 warn(f"Failed to capture new special columns for {self}:\n{capture_msg}") 452 453 if debug: 454 dprint( 455 "DataFrame to sync:\n" 456 + ( 457 str(df)[:255] 458 + '...' 459 if len(str(df)) >= 256 460 else str(df) 461 ), 462 **kw 463 ) 464 465 ### if force, continue to sync until success 466 return_tuple = False, f"Did not sync {p}." 467 run = True 468 _retries = 1 469 while run: 470 with Venv(get_connector_plugin(self.instance_connector)): 471 return_tuple = p.instance_connector.sync_pipe( 472 pipe=p, 473 df=df, 474 debug=debug, 475 **kw 476 ) 477 _retries += 1 478 run = (not return_tuple[0]) and force and _retries <= retries 479 if run and debug: 480 dprint(f"Syncing failed for {p}. Attempt ( {_retries} / {retries} )", **kw) 481 dprint(f"Sleeping for {min_seconds} seconds...", **kw) 482 time.sleep(min_seconds) 483 if _retries > retries: 484 warn( 485 f"Unable to sync {p} within {retries} attempt" + 486 ("s" if retries != 1 else "") + "!" 487 ) 488 489 ### CHECKPOINT: Finished syncing. 490 _checkpoint(**kw) 491 p._invalidate_cache(debug=debug) 492 493 ### Automatically apply a compression policy if the pipe is configured for compression. 494 if return_tuple[0] and p.parameters.get('compress', False): 495 if hasattr(p.instance_connector, 'apply_compression_policy'): 496 try: 497 with Venv(get_connector_plugin(p.instance_connector)): 498 compress_success, compress_msg = ( 499 p.instance_connector.apply_compression_policy(p, debug=debug) 500 ) 501 if not compress_success and debug: 502 dprint(f"Could not apply compression policy to {p}:\n{compress_msg}") 503 except Exception as compress_e: 504 warn( 505 f"Failed to apply compression policy to {p}:\n{compress_e}", 506 stack=False, 507 ) 508 509 return return_tuple 510 511 if blocking: 512 return _sync(self, df=df) 513 514 from meerschaum.utils.threading import Thread 515 def default_callback(result_tuple: SuccessTuple): 516 dprint(f"Asynchronous result from {self}: {result_tuple}", **kw) 517 518 def default_error_callback(x: Exception): 519 dprint(f"Error received for {self}: {x}", **kw) 520 521 if callback is None and debug: 522 callback = default_callback 523 if error_callback is None and debug: 524 error_callback = default_error_callback 525 try: 526 thread = Thread( 527 target=_sync, 528 args=(self,), 529 kwargs={'df': df}, 530 daemon=False, 531 callback=callback, 532 error_callback=error_callback, 533 ) 534 thread.start() 535 except Exception as e: 536 self._invalidate_cache(debug=debug) 537 return False, str(e) 538 539 self._invalidate_cache(debug=debug) 540 return True, f"Spawned asyncronous sync for {self}."
Fetch new data from the source and update the pipe's table with new data.
Get new remote data via fetch, get existing data in the same time period, and merge the two, only keeping the unseen data.
Parameters
- df (Union[None, pd.DataFrame, Dict[str, List[Any]]], default None):
An optional DataFrame to sync into the pipe. Defaults to
None. Ifdfis a string, it will be parsed viameerschaum.utils.dataframe.parse_simple_lines(). - begin (Union[datetime, int, str, None], default ''): Optionally specify the earliest datetime to search for data.
- end (Union[datetime, int, str, None], default None): Optionally specify the latest datetime to search for data.
- force (bool, default False):
If
True, keep trying to sync untulretriesattempts. - retries (int, default 10):
If
force, how many attempts to try syncing before declaring failure. - min_seconds (Union[int, float], default 1):
If
force, how many seconds to sleep between retries. Defaults to1. - check_existing (bool, default True):
If
True, pull and diff with existing data from the pipe. - enforce_dtypes (bool, default True):
If
True, enforce dtypes on incoming data. Set this toFalseif the incoming rows are expected to be of the correct dtypes. - blocking (bool, default True):
If
True, wait for sync to finish and return its result, otherwise asyncronously sync (oxymoron?) and return success. Defaults toTrue. Only intended for specific scenarios. - workers (Optional[int], default None):
If provided and the instance connector is thread-safe
(
pipe.instance_connector.IS_THREAD_SAFE is True), limit concurrent sync to this many threads. - callback (Optional[Callable[[Tuple[bool, str]], Any]], default None):
Callback function which expects a SuccessTuple as input.
Only applies when
blocking=False. - error_callback (Optional[Callable[[Exception], Any]], default None):
Callback function which expects an Exception as input.
Only applies when
blocking=False. - chunksize (int, default -1):
Specify the number of rows to sync per chunk.
If
-1, resort to system configuration (default is900). AchunksizeofNonewill sync all rows in one transaction. - sync_chunks (bool, default True): If possible, sync chunks while fetching them into memory.
- debug (bool, default False): Verbosity toggle. Defaults to False.
Returns
- A
SuccessTupleof success (bool) and message (str).
543def get_sync_time( 544 self, 545 params: Optional[Dict[str, Any]] = None, 546 newest: bool = True, 547 apply_backtrack_interval: bool = False, 548 remote: bool = False, 549 round_down: bool = False, 550 debug: bool = False 551) -> Union['datetime', int, None]: 552 """ 553 Get the most recent datetime value for a Pipe. 554 555 Parameters 556 ---------- 557 params: Optional[Dict[str, Any]], default None 558 Dictionary to build a WHERE clause for a specific column. 559 See `meerschaum.utils.sql.build_where`. 560 561 newest: bool, default True 562 If `True`, get the most recent datetime (honoring `params`). 563 If `False`, get the oldest datetime (`ASC` instead of `DESC`). 564 565 apply_backtrack_interval: bool, default False 566 If `True`, subtract the backtrack interval from the sync time. 567 568 remote: bool, default False 569 If `True` and the instance connector supports it, return the sync time 570 for the remote table definition. 571 572 round_down: bool, default False 573 If `True`, round down the datetime value to the nearest minute. 574 575 debug: bool, default False 576 Verbosity toggle. 577 578 Returns 579 ------- 580 A `datetime` or int, if the pipe exists, otherwise `None`. 581 582 """ 583 from meerschaum.utils.venv import Venv 584 from meerschaum.connectors import get_connector_plugin 585 from meerschaum.utils.misc import filter_keywords 586 from meerschaum.utils.dtypes import round_time 587 from meerschaum.utils.warnings import warn 588 589 if not self.columns.get('datetime', None): 590 return None 591 592 connector = self.instance_connector if not remote else self.connector 593 if isinstance(connector, str) or connector is None: 594 return None 595 596 with Venv(get_connector_plugin(connector)): 597 if not hasattr(connector, 'get_sync_time'): 598 warn( 599 f"Connectors of type '{connector.type}' " 600 "do not implement `get_sync_time().", 601 stack=False, 602 ) 603 return None 604 sync_time = connector.get_sync_time( 605 self, 606 **filter_keywords( 607 connector.get_sync_time, 608 params=params, 609 newest=newest, 610 remote=remote, 611 debug=debug, 612 ) 613 ) 614 615 if round_down and isinstance(sync_time, datetime): 616 sync_time = round_time(sync_time, timedelta(minutes=1)) 617 618 if apply_backtrack_interval and sync_time is not None: 619 backtrack_interval = self.get_backtrack_interval(debug=debug) 620 try: 621 sync_time -= backtrack_interval 622 except Exception as e: 623 warn(f"Failed to apply backtrack interval:\n{e}") 624 625 return self.parse_date_bounds(sync_time)
Get the most recent datetime value for a Pipe.
Parameters
- params (Optional[Dict[str, Any]], default None):
Dictionary to build a WHERE clause for a specific column.
See
meerschaum.utils.sql.build_where. - newest (bool, default True):
If
True, get the most recent datetime (honoringparams). IfFalse, get the oldest datetime (ASCinstead ofDESC). - apply_backtrack_interval (bool, default False):
If
True, subtract the backtrack interval from the sync time. - remote (bool, default False):
If
Trueand the instance connector supports it, return the sync time for the remote table definition. - round_down (bool, default False):
If
True, round down the datetime value to the nearest minute. - debug (bool, default False): Verbosity toggle.
Returns
- A
datetimeor int, if the pipe exists, otherwiseNone.
628def exists( 629 self, 630 debug: bool = False 631) -> bool: 632 """ 633 See if a Pipe's table exists. 634 635 Parameters 636 ---------- 637 debug: bool, default False 638 Verbosity toggle. 639 640 Returns 641 ------- 642 A `bool` corresponding to whether a pipe's underlying table exists. 643 644 """ 645 from meerschaum.utils.venv import Venv 646 from meerschaum.connectors import get_connector_plugin 647 from meerschaum.utils.debug import dprint 648 from meerschaum.utils.dtypes import get_current_timestamp 649 now = get_current_timestamp('ms', as_int=True) / 1000 650 cache_seconds = mrsm.get_config('pipes', 'sync', 'exists_cache_seconds') 651 652 _exists = self._get_cached_value('_exists', debug=debug) 653 if _exists: 654 exists_timestamp = self._get_cached_value('_exists_timestamp', debug=debug) 655 if exists_timestamp is not None: 656 delta = now - exists_timestamp 657 if delta < cache_seconds: 658 if debug: 659 dprint(f"Returning cached `exists` for {self} ({round(delta, 2)} seconds old).") 660 return _exists 661 662 with Venv(get_connector_plugin(self.instance_connector)): 663 _exists = ( 664 self.instance_connector.pipe_exists(pipe=self, debug=debug) 665 if hasattr(self.instance_connector, 'pipe_exists') 666 else False 667 ) 668 669 self._cache_value('_exists', _exists, debug=debug) 670 self._cache_value('_exists_timestamp', now, debug=debug) 671 return _exists
See if a Pipe's table exists.
Parameters
- debug (bool, default False): Verbosity toggle.
Returns
- A
boolcorresponding to whether a pipe's underlying table exists.
674def filter_existing( 675 self, 676 df: 'pd.DataFrame', 677 safe_copy: bool = True, 678 date_bound_only: bool = False, 679 include_unchanged_columns: bool = False, 680 enforce_dtypes: bool = False, 681 chunksize: Optional[int] = -1, 682 debug: bool = False, 683 **kw 684) -> Tuple['pd.DataFrame', 'pd.DataFrame', 'pd.DataFrame']: 685 """ 686 Inspect a dataframe and filter out rows which already exist in the pipe. 687 688 Parameters 689 ---------- 690 df: 'pd.DataFrame' 691 The dataframe to inspect and filter. 692 A `polars.DataFrame` or `LazyFrame` is accepted, 693 in which case the returned dataframes are `polars.DataFrame`. 694 695 safe_copy: bool, default True 696 If `True`, create a copy before comparing and modifying the dataframes. 697 Setting to `False` may mutate the DataFrames. 698 See `meerschaum.utils.dataframe.filter_unseen_df`. 699 700 date_bound_only: bool, default False 701 If `True`, only use the datetime index to fetch the sample dataframe. 702 703 include_unchanged_columns: bool, default False 704 If `True`, include the backtrack columns which haven't changed in the update dataframe. 705 This is useful if you can't update individual keys. 706 707 enforce_dtypes: bool, default False 708 If `True`, ensure the given and intermediate dataframes are enforced to the correct dtypes. 709 Setting `enforce_dtypes=True` may impact performance. 710 711 chunksize: Optional[int], default -1 712 The `chunksize` used when fetching existing data. 713 714 debug: bool, default False 715 Verbosity toggle. 716 717 Returns 718 ------- 719 A tuple of three DataFrames: unseen, update, and delta. 720 These are Polars DataFrames if `df` is a Polars DataFrame or LazyFrame, else Pandas. 721 """ 722 from meerschaum.utils.warnings import warn 723 from meerschaum.utils.debug import dprint 724 from meerschaum.utils.packages import attempt_import, import_pandas 725 from meerschaum.utils.dataframe import ( 726 filter_unseen_df, 727 add_missing_cols_to_df, 728 get_unhashable_cols, 729 get_json_cols, 730 to_pandas, 731 to_polars, 732 ) 733 from meerschaum.utils.dtypes import ( 734 to_pandas_dtype, 735 none_if_null, 736 ) 737 from meerschaum.config import get_config 738 pd = import_pandas() 739 pandas = attempt_import('pandas') 740 ### Polars frames are filtered natively when possible; 741 ### otherwise they are converted up-front and converted back on the way out. 742 input_is_polars = df is not None and df.__class__.__module__.split('.')[0] == 'polars' 743 if input_is_polars: 744 polars_dfs = self._filter_existing_polars( 745 df, 746 safe_copy=safe_copy, 747 date_bound_only=date_bound_only, 748 include_unchanged_columns=include_unchanged_columns, 749 enforce_dtypes=enforce_dtypes, 750 chunksize=chunksize, 751 debug=debug, 752 **kw 753 ) 754 if polars_dfs is not None: 755 return polars_dfs 756 757 df = to_pandas(df) 758 759 def _return_dfs(unseen, update, delta): 760 if not input_is_polars: 761 return unseen, update, delta 762 return tuple( 763 to_polars(_df, json_cols=get_json_cols(_df)) 764 for _df in (unseen, update, delta) 765 ) 766 767 if enforce_dtypes or 'dataframe' not in str(type(df)).lower(): 768 df = self.enforce_dtypes(df, chunksize=chunksize, debug=debug) 769 is_dask = hasattr(df, '__module__') and 'dask' in df.__module__ 770 if is_dask: 771 dd = attempt_import('dask.dataframe') 772 merge = dd.merge 773 NA = pandas.NA 774 else: 775 merge = pd.merge 776 NA = pd.NA 777 778 parameters = self.parameters 779 pipe_columns = self.columns 780 primary_key = pipe_columns.get('primary', None) 781 dt_col = pipe_columns.get('datetime', None) 782 dt_type = parameters.get('dtypes', {}).get(dt_col, 'datetime') if dt_col else None 783 autoincrement = parameters.get('autoincrement', False) 784 autotime = parameters.get('autotime', False) 785 786 if primary_key and autoincrement and df is not None and primary_key in df.columns: 787 if safe_copy: 788 df = df.copy() 789 safe_copy = False 790 if df[primary_key].isnull().all(): 791 del df[primary_key] 792 _ = self.columns.pop(primary_key, None) 793 794 if dt_col and autotime and df is not None and dt_col in df.columns: 795 if safe_copy: 796 df = df.copy() 797 safe_copy = False 798 if df[dt_col].isnull().all(): 799 del df[dt_col] 800 _ = self.columns.pop(dt_col, None) 801 802 def get_empty_df(): 803 empty_df = pd.DataFrame([]) 804 dtypes = dict(df.dtypes) if df is not None else {} 805 dtypes.update(self.dtypes) if self.enforce else {} 806 pd_dtypes = { 807 col: to_pandas_dtype(str(typ)) 808 for col, typ in dtypes.items() 809 } 810 return add_missing_cols_to_df(empty_df, pd_dtypes) 811 812 if df is None: 813 empty_df = get_empty_df() 814 return _return_dfs(empty_df, empty_df, empty_df) 815 816 if (df.empty if not is_dask else len(df) == 0): 817 return _return_dfs(df, df, df) 818 819 if autoincrement and primary_key == dt_col and dt_col not in df.columns: 820 if enforce_dtypes: 821 df = self.enforce_dtypes(df, chunksize=chunksize, debug=debug) 822 return _return_dfs(df, get_empty_df(), df) 823 824 if autotime and dt_col and dt_col not in df.columns: 825 if enforce_dtypes: 826 df = self.enforce_dtypes(df, chunksize=chunksize, debug=debug) 827 return _return_dfs(df, get_empty_df(), df) 828 829 try: 830 min_dt_val = df[dt_col].min(skipna=True) if dt_col and dt_col in df.columns else None 831 max_dt_val = df[dt_col].max(skipna=True) if dt_col and dt_col in df.columns else None 832 if is_dask: 833 min_dt_val = min_dt_val.compute() if min_dt_val is not None else None 834 max_dt_val = max_dt_val.compute() if max_dt_val is not None else None 835 except Exception: 836 min_dt_val, max_dt_val = None, None 837 838 ### `begin` is the oldest data in the new dataframe, `end` is the newest. 839 begin, end = _get_delta_bounds(min_dt_val, max_dt_val, dt_type, dt_col) 840 841 unique_index_vals = { 842 col: df[col].unique() 843 for col in (pipe_columns.values() if not primary_key else [primary_key]) 844 if col in df.columns and col != dt_col 845 } if not date_bound_only else {} 846 unique_index_lens = { 847 col: len(unique_vals) 848 for col, unique_vals in unique_index_vals.items() 849 } if not date_bound_only else {} 850 filter_params_index_limit = get_config('pipes', 'sync', 'filter_params_index_limit') 851 _ = kw.pop('params', None) 852 params = { 853 col: [ 854 none_if_null(val) 855 for val in unique_vals 856 ] 857 for col, unique_vals in unique_index_vals.items() 858 if unique_index_lens[col] <= filter_params_index_limit 859 } if not date_bound_only else {} 860 861 if debug: 862 dprint( 863 ( 864 f"Looking at data between '{begin}' and '{end}' with index value lengths:\n" 865 f"{json.dumps(unique_index_lens, indent=4)}\n" 866 ), 867 **kw 868 ) 869 870 backtrack_df = self.get_data( 871 begin=begin, 872 end=end, 873 chunksize=chunksize, 874 params=params, 875 debug=debug, 876 **kw 877 ) 878 if backtrack_df is None: 879 if debug: 880 dprint(f"No backtrack data was found for {self}.") 881 return _return_dfs(df, get_empty_df(), df) 882 883 if enforce_dtypes: 884 backtrack_df = self.enforce_dtypes(backtrack_df, chunksize=chunksize, debug=debug) 885 886 if debug: 887 dprint(f"Existing data for {self}:\n" + str(backtrack_df), **kw) 888 dprint(f"Existing dtypes for {self}:\n" + str(backtrack_df.dtypes)) 889 890 ### Separate new rows from changed ones. 891 on_cols = [ 892 col 893 for col_key, col in pipe_columns.items() 894 if ( 895 col 896 and 897 col_key != 'value' 898 and col in backtrack_df.columns 899 ) 900 ] if not primary_key else [primary_key] 901 902 self_dtypes = self.get_dtypes(debug=debug) if self.enforce else {} 903 on_cols_dtypes = { 904 col: to_pandas_dtype(typ) 905 for col, typ in self_dtypes.items() 906 if col in on_cols 907 } 908 909 ### Detect changes between the old target and new source dataframes. 910 delta_df = add_missing_cols_to_df( 911 filter_unseen_df( 912 backtrack_df, 913 df, 914 dtypes={ 915 col: to_pandas_dtype(typ) 916 for col, typ in self_dtypes.items() 917 }, 918 safe_copy=safe_copy, 919 coerce_mixed_numerics=(not self.static), 920 debug=debug 921 ), 922 on_cols_dtypes, 923 ) 924 if enforce_dtypes: 925 delta_df = self.enforce_dtypes(delta_df, chunksize=chunksize, debug=debug) 926 927 for col, typ in on_cols_dtypes.items(): 928 delta_df[col] = delta_df[col].astype(typ) 929 backtrack_df[col] = backtrack_df[col].astype(typ) 930 931 ### Cast dicts or lists to strings so we can merge. 932 serializer = functools.partial(json.dumps, sort_keys=True, separators=(',', ':'), default=str) 933 934 def deserializer(x): 935 return json.loads(x) if isinstance(x, str) else x 936 937 unhashable_delta_cols = get_unhashable_cols(delta_df) 938 unhashable_backtrack_cols = get_unhashable_cols(backtrack_df) 939 for col in unhashable_delta_cols: 940 delta_df[col] = delta_df[col].apply(serializer) 941 for col in unhashable_backtrack_cols: 942 backtrack_df[col] = backtrack_df[col].apply(serializer) 943 casted_cols = set(unhashable_delta_cols + unhashable_backtrack_cols) 944 945 joined_df = merge( 946 delta_df.infer_objects().fillna(NA), 947 backtrack_df.infer_objects().fillna(NA), 948 how='left', 949 on=on_cols, 950 indicator=True, 951 suffixes=('', '_old'), 952 ) if on_cols else delta_df 953 for col in casted_cols: 954 if col in joined_df.columns: 955 joined_df[col] = joined_df[col].apply(deserializer) 956 if col in delta_df.columns: 957 delta_df[col] = delta_df[col].apply(deserializer) 958 959 ### Determine which rows are completely new. 960 new_rows_mask = (joined_df['_merge'] == 'left_only') if on_cols else None 961 cols = list(delta_df.columns) 962 963 unseen_df = ( 964 joined_df[new_rows_mask][cols].reset_index(drop=True) 965 ) if on_cols else delta_df 966 967 ### Rows that have already been inserted but values have changed. 968 update_df = ( 969 joined_df[~new_rows_mask][cols].reset_index(drop=True) 970 ) if on_cols else get_empty_df() 971 972 if include_unchanged_columns and on_cols: 973 unchanged_backtrack_cols = [ 974 col 975 for col in backtrack_df.columns 976 if col in on_cols or col not in update_df.columns 977 ] 978 if enforce_dtypes: 979 update_df = self.enforce_dtypes(update_df, chunksize=chunksize, debug=debug) 980 update_df = merge( 981 backtrack_df[unchanged_backtrack_cols], 982 update_df, 983 how='inner', 984 on=on_cols, 985 ) 986 987 return _return_dfs(unseen_df, update_df, delta_df)
Inspect a dataframe and filter out rows which already exist in the pipe.
Parameters
- df ('pd.DataFrame'):
The dataframe to inspect and filter.
A
polars.DataFrameorLazyFrameis accepted, in which case the returned dataframes arepolars.DataFrame. - safe_copy (bool, default True):
If
True, create a copy before comparing and modifying the dataframes. Setting toFalsemay mutate the DataFrames. Seemeerschaum.utils.dataframe.filter_unseen_df. - date_bound_only (bool, default False):
If
True, only use the datetime index to fetch the sample dataframe. - include_unchanged_columns (bool, default False):
If
True, include the backtrack columns which haven't changed in the update dataframe. This is useful if you can't update individual keys. - enforce_dtypes (bool, default False):
If
True, ensure the given and intermediate dataframes are enforced to the correct dtypes. Settingenforce_dtypes=Truemay impact performance. - chunksize (Optional[int], default -1):
The
chunksizeused when fetching existing data. - debug (bool, default False): Verbosity toggle.
Returns
A tuple of three DataFrames (unseen, update, and delta.):
These are Polars DataFrames if
dfis a Polars DataFrame or LazyFrame, else Pandas.
1381def get_num_workers(self, workers: Optional[int] = None) -> int: 1382 """ 1383 Get the number of workers to use for concurrent syncs. 1384 1385 Parameters 1386 ---------- 1387 The number of workers passed via `--workers`. 1388 1389 Returns 1390 ------- 1391 The number of workers, capped for safety. 1392 """ 1393 is_thread_safe = getattr(self.instance_connector, 'IS_THREAD_SAFE', False) 1394 if not is_thread_safe: 1395 return 1 1396 1397 engine_pool_size = ( 1398 self.instance_connector.engine.pool.size() 1399 if self.instance_connector.type == 'sql' 1400 else None 1401 ) 1402 current_num_threads = threading.active_count() 1403 current_num_connections = ( 1404 self.instance_connector.engine.pool.checkedout() 1405 if engine_pool_size is not None 1406 else current_num_threads 1407 ) 1408 desired_workers = ( 1409 min(workers or engine_pool_size, engine_pool_size) 1410 if engine_pool_size is not None 1411 else workers 1412 ) 1413 if desired_workers is None: 1414 desired_workers = (multiprocessing.cpu_count() if is_thread_safe else 1) 1415 1416 return max( 1417 (desired_workers - current_num_connections), 1418 1, 1419 )
Get the number of workers to use for concurrent syncs.
Parameters
- The number of workers passed via
--workers.
Returns
- The number of workers, capped for safety.
19def verify( 20 self, 21 begin: Union[datetime, int, None] = None, 22 end: Union[datetime, int, None] = None, 23 params: Optional[Dict[str, Any]] = None, 24 chunk_interval: Union[timedelta, int, None] = None, 25 bounded: Optional[bool] = None, 26 deduplicate: bool = False, 27 workers: Optional[int] = None, 28 batchsize: Optional[int] = None, 29 skip_chunks_with_greater_rowcounts: bool = False, 30 check_rowcounts_only: bool = False, 31 debug: bool = False, 32 **kwargs: Any 33) -> SuccessTuple: 34 """ 35 Verify the contents of the pipe by resyncing its interval. 36 37 Parameters 38 ---------- 39 begin: Union[datetime, int, None], default None 40 If specified, only verify rows greater than or equal to this value. 41 42 end: Union[datetime, int, None], default None 43 If specified, only verify rows less than this value. 44 45 chunk_interval: Union[timedelta, int, None], default None 46 If provided, use this as the size of the chunk boundaries. 47 Default to the value set in `pipe.parameters['verify']['chunk_minutes']` (43200 — 30 days). 48 49 bounded: Optional[bool], default None 50 If `True`, do not verify older than the oldest sync time or newer than the newest. 51 If `False`, verify unbounded syncs outside of the new and old sync times. 52 The default behavior (`None`) is to bound only if a bound interval is set 53 (e.g. `pipe.parameters['verify']['bound_days']`). 54 55 deduplicate: bool, default False 56 If `True`, deduplicate the pipe's table after the verification syncs. 57 58 workers: Optional[int], default None 59 If provided, limit the verification to this many threads. 60 Use a value of `1` to sync chunks in series. 61 62 batchsize: Optional[int], default None 63 If provided, sync this many chunks in parallel. 64 Defaults to `Pipe.get_num_workers()`. 65 66 skip_chunks_with_greater_rowcounts: bool, default False 67 If `True`, compare the rowcounts for a chunk and skip syncing if the pipe's 68 chunk rowcount equals or exceeds the remote's rowcount. 69 70 check_rowcounts_only: bool, default False 71 If `True`, only compare rowcounts and print chunks which are out-of-sync. 72 73 debug: bool, default False 74 Verbosity toggle. 75 76 kwargs: Any 77 All keyword arguments are passed to `pipe.sync()`. 78 79 Returns 80 ------- 81 A SuccessTuple indicating whether the pipe was successfully resynced. 82 """ 83 from meerschaum.utils.pool import get_pool 84 from meerschaum.utils.formatting import make_header 85 from meerschaum.utils.misc import interval_str 86 workers = self.get_num_workers(workers) 87 check_rowcounts = skip_chunks_with_greater_rowcounts or check_rowcounts_only 88 89 ### Skip configured bounding in parameters 90 ### if `bounded` is explicitly `False`. 91 bound_time = ( 92 self.get_bound_time(debug=debug) 93 if bounded is not False 94 else None 95 ) 96 if bounded is None: 97 bounded = bound_time is not None 98 99 if bounded and begin is None: 100 begin = ( 101 bound_time 102 if bound_time is not None 103 else self.get_sync_time(newest=False, debug=debug) 104 ) 105 if begin is None: 106 remote_oldest_sync_time = self.get_sync_time(newest=False, remote=True, debug=debug) 107 begin = remote_oldest_sync_time 108 if bounded and end is None: 109 end = self.get_sync_time(newest=True, debug=debug) 110 if end is None: 111 remote_newest_sync_time = self.get_sync_time(newest=True, remote=True, debug=debug) 112 end = remote_newest_sync_time 113 if end is not None: 114 end += ( 115 timedelta(minutes=1) 116 if hasattr(end, 'tzinfo') 117 else 1 118 ) 119 120 begin, end = self.parse_date_bounds(begin, end) 121 cannot_determine_bounds = bounded and begin is None and end is None 122 123 if cannot_determine_bounds and not check_rowcounts_only: 124 warn(f"Cannot determine sync bounds for {self}. Syncing instead...", stack=False) 125 sync_success, sync_msg = self.sync( 126 begin=begin, 127 end=end, 128 params=params, 129 workers=workers, 130 debug=debug, 131 **kwargs 132 ) 133 if not sync_success: 134 return sync_success, sync_msg 135 136 if deduplicate: 137 return self.deduplicate( 138 begin=begin, 139 end=end, 140 params=params, 141 workers=workers, 142 debug=debug, 143 **kwargs 144 ) 145 return sync_success, sync_msg 146 147 chunk_interval = self.get_chunk_interval(chunk_interval, debug=debug) 148 chunk_bounds = self.get_chunk_bounds( 149 begin=begin, 150 end=end, 151 chunk_interval=chunk_interval, 152 bounded=bounded, 153 align=True, 154 debug=debug, 155 ) 156 157 ### Consider it a success if no chunks need to be verified. 158 if not chunk_bounds: 159 if deduplicate: 160 return self.deduplicate( 161 begin=begin, 162 end=end, 163 params=params, 164 workers=workers, 165 debug=debug, 166 **kwargs 167 ) 168 return True, f"Could not determine chunks between '{begin}' and '{end}'; nothing to do." 169 170 begin_to_print = ( 171 begin 172 if begin is not None 173 else ( 174 chunk_bounds[0][0] 175 if bounded 176 else chunk_bounds[0][1] 177 ) 178 ) 179 end_to_print = ( 180 end 181 if end is not None 182 else ( 183 chunk_bounds[-1][1] 184 if bounded 185 else chunk_bounds[-1][0] 186 ) 187 ) 188 message_header = f"{begin_to_print} - {end_to_print}" 189 max_chunks_syncs = mrsm.get_config('pipes', 'verify', 'max_chunks_syncs') 190 191 info( 192 f"Verifying {self}:\n " 193 + ("Syncing" if not check_rowcounts_only else "Checking") 194 + f" {len(chunk_bounds)} chunk" 195 + ('s' if len(chunk_bounds) != 1 else '') 196 + f" ({'un' if not bounded else ''}bounded)" 197 + f" of size '{interval_str(chunk_interval)}'" 198 + f" between '{begin_to_print}' and '{end_to_print}'.\n" 199 ) 200 201 ### Dictionary of the form bounds -> success_tuple, e.g.: 202 ### { 203 ### (2023-01-01, 2023-01-02): (True, "Success") 204 ### } 205 bounds_success_tuples = {} 206 def process_chunk_bounds( 207 chunk_begin_and_end: Tuple[ 208 Union[int, datetime], 209 Union[int, datetime] 210 ], 211 _workers: Optional[int] = 1, 212 ): 213 if chunk_begin_and_end in bounds_success_tuples: 214 return chunk_begin_and_end, bounds_success_tuples[chunk_begin_and_end] 215 216 chunk_begin, chunk_end = chunk_begin_and_end 217 do_sync = True 218 chunk_success, chunk_msg = False, "Did not sync chunk." 219 if check_rowcounts: 220 existing_rowcount = self.get_rowcount(begin=chunk_begin, end=chunk_end, debug=debug) 221 remote_rowcount = self.get_rowcount( 222 begin=chunk_begin, 223 end=chunk_end, 224 remote=True, 225 debug=debug, 226 ) 227 checked_rows_str = ( 228 f"checked {existing_rowcount:,} row" 229 + ("s" if existing_rowcount != 1 else '') 230 + f" vs {remote_rowcount:,} remote" 231 ) 232 if ( 233 existing_rowcount is not None 234 and remote_rowcount is not None 235 and existing_rowcount >= remote_rowcount 236 ): 237 do_sync = False 238 chunk_success, chunk_msg = True, ( 239 "Row-count is up-to-date " 240 f"({checked_rows_str})." 241 ) 242 elif check_rowcounts_only: 243 do_sync = False 244 chunk_success, chunk_msg = True, ( 245 f"Row-counts are out-of-sync ({checked_rows_str})." 246 ) 247 248 num_syncs = 0 249 while num_syncs < max_chunks_syncs: 250 chunk_success, chunk_msg = self.sync( 251 begin=chunk_begin, 252 end=chunk_end, 253 params=params, 254 workers=_workers, 255 debug=debug, 256 **kwargs 257 ) if do_sync else (chunk_success, chunk_msg) 258 if chunk_success: 259 break 260 num_syncs += 1 261 time.sleep(num_syncs**2) 262 chunk_msg = chunk_msg.strip() 263 if ' - ' not in chunk_msg: 264 chunk_label = f"{chunk_begin} - {chunk_end}" 265 chunk_msg = f'Verified chunk for {self}:\n{chunk_label}\n{chunk_msg}' 266 mrsm.pprint((chunk_success, chunk_msg)) 267 268 return chunk_begin_and_end, (chunk_success, chunk_msg) 269 270 ### If we have more than one chunk, attempt to sync the first one and return if its fails. 271 if len(chunk_bounds) > 1: 272 first_chunk_bounds = chunk_bounds[0] 273 first_label = f"{first_chunk_bounds[0]} - {first_chunk_bounds[1]}" 274 info(f"Verifying first chunk for {self}:\n {first_label}") 275 ( 276 (first_begin, first_end), 277 (first_success, first_msg) 278 ) = process_chunk_bounds(first_chunk_bounds, _workers=workers) 279 if not first_success: 280 return ( 281 first_success, 282 f"\n{first_label}\n" 283 + f"Failed to sync first chunk:\n{first_msg}" 284 ) 285 bounds_success_tuples[first_chunk_bounds] = (first_success, first_msg) 286 info(f"Completed first chunk for {self}:\n {first_label}\n") 287 chunk_bounds = chunk_bounds[1:] 288 289 pool = get_pool(workers=workers) 290 batches = self.get_chunk_bounds_batches(chunk_bounds, batchsize=batchsize, workers=workers) 291 292 def process_batch( 293 batch_chunk_bounds: Tuple[ 294 Tuple[Union[datetime, int, None], Union[datetime, int, None]], 295 ... 296 ] 297 ): 298 _batch_begin = batch_chunk_bounds[0][0] 299 _batch_end = batch_chunk_bounds[-1][-1] 300 batch_message_header = f"{_batch_begin} - {_batch_end}" 301 302 if check_rowcounts_only: 303 info(f"Checking row-counts for batch bounds:\n {batch_message_header}") 304 _, (batch_init_success, batch_init_msg) = process_chunk_bounds( 305 (_batch_begin, _batch_end) 306 ) 307 mrsm.pprint((batch_init_success, batch_init_msg)) 308 if batch_init_success and 'up-to-date' in batch_init_msg: 309 info("Entire batch is up-to-date.") 310 return batch_init_success, batch_init_msg 311 312 batch_bounds_success_tuples = dict(pool.map(process_chunk_bounds, batch_chunk_bounds)) 313 bounds_success_tuples.update(batch_bounds_success_tuples) 314 batch_bounds_success_bools = { 315 bounds: tup[0] 316 for bounds, tup in batch_bounds_success_tuples.items() 317 } 318 319 if all(batch_bounds_success_bools.values()): 320 msg = get_chunks_success_message( 321 batch_bounds_success_tuples, 322 header=batch_message_header, 323 check_rowcounts_only=check_rowcounts_only, 324 ) 325 if deduplicate: 326 deduplicate_success, deduplicate_msg = self.deduplicate( 327 begin=_batch_begin, 328 end=_batch_end, 329 params=params, 330 workers=workers, 331 debug=debug, 332 **kwargs 333 ) 334 return deduplicate_success, msg + '\n\n' + deduplicate_msg 335 return True, msg 336 337 batch_chunk_bounds_to_resync = [ 338 bounds 339 for bounds, success in zip(batch_chunk_bounds, batch_bounds_success_bools) 340 if not success 341 ] 342 batch_bounds_to_print = [ 343 f"{bounds[0]} - {bounds[1]}" 344 for bounds in batch_chunk_bounds_to_resync 345 ] 346 if batch_bounds_to_print: 347 warn( 348 "Will resync the following failed chunks:\n " 349 + '\n '.join(batch_bounds_to_print), 350 stack=False, 351 ) 352 353 retry_bounds_success_tuples = dict(pool.map( 354 process_chunk_bounds, 355 batch_chunk_bounds_to_resync 356 )) 357 batch_bounds_success_tuples.update(retry_bounds_success_tuples) 358 bounds_success_tuples.update(retry_bounds_success_tuples) 359 retry_bounds_success_bools = { 360 bounds: tup[0] 361 for bounds, tup in retry_bounds_success_tuples.items() 362 } 363 364 if all(retry_bounds_success_bools.values()): 365 chunks_message = ( 366 get_chunks_success_message( 367 batch_bounds_success_tuples, 368 header=batch_message_header, 369 check_rowcounts_only=check_rowcounts_only, 370 ) + f"\nRetried {len(batch_chunk_bounds_to_resync)} chunk" + ( 371 's' 372 if len(batch_chunk_bounds_to_resync) != 1 373 else '' 374 ) + "." 375 ) 376 if deduplicate: 377 deduplicate_success, deduplicate_msg = self.deduplicate( 378 begin=_batch_begin, 379 end=_batch_end, 380 params=params, 381 workers=workers, 382 debug=debug, 383 **kwargs 384 ) 385 return deduplicate_success, chunks_message + '\n\n' + deduplicate_msg 386 return True, chunks_message 387 388 batch_chunks_message = get_chunks_success_message( 389 batch_bounds_success_tuples, 390 header=batch_message_header, 391 check_rowcounts_only=check_rowcounts_only, 392 ) 393 if deduplicate: 394 deduplicate_success, deduplicate_msg = self.deduplicate( 395 begin=begin, 396 end=end, 397 params=params, 398 workers=workers, 399 debug=debug, 400 **kwargs 401 ) 402 return deduplicate_success, batch_chunks_message + '\n\n' + deduplicate_msg 403 return False, batch_chunks_message 404 405 num_batches = len(batches) 406 for batch_i, batch in enumerate(batches): 407 batch_begin = batch[0][0] 408 batch_end = batch[-1][-1] 409 batch_counter_str = f"({(batch_i + 1):,}/{num_batches:,})" 410 batch_label = f"batch {batch_counter_str}:\n{batch_begin} - {batch_end}" 411 retry_failed_batch = True 412 try: 413 for_self = 'for ' + str(self) 414 batch_label_str = batch_label.replace(':\n', ' ' + for_self + '...\n ') 415 info(f"Verifying {batch_label_str}\n") 416 batch_success, batch_msg = process_batch(batch) 417 except (KeyboardInterrupt, Exception) as e: 418 batch_success = False 419 batch_msg = str(e) 420 retry_failed_batch = False 421 422 batch_msg_to_print = ( 423 f"{make_header('Completed batch ' + batch_counter_str + ':', left_pad=0)}\n{batch_msg}" 424 ) 425 mrsm.pprint((batch_success, batch_msg_to_print)) 426 427 if not batch_success and retry_failed_batch: 428 info(f"Retrying batch {batch_counter_str}...") 429 retry_batch_success, retry_batch_msg = process_batch(batch) 430 retry_batch_msg_to_print = ( 431 f"Retried {make_header('batch ' + batch_label, left_pad=0)}\n{retry_batch_msg}" 432 ) 433 mrsm.pprint((retry_batch_success, retry_batch_msg_to_print)) 434 435 batch_success = retry_batch_success 436 batch_msg = retry_batch_msg 437 438 if not batch_success: 439 return False, f"Failed to verify {batch_label}:\n\n{batch_msg}" 440 441 chunks_message = get_chunks_success_message( 442 bounds_success_tuples, 443 header=message_header, 444 check_rowcounts_only=check_rowcounts_only, 445 ) 446 return True, chunks_message
Verify the contents of the pipe by resyncing its interval.
Parameters
- begin (Union[datetime, int, None], default None): If specified, only verify rows greater than or equal to this value.
- end (Union[datetime, int, None], default None): If specified, only verify rows less than this value.
- chunk_interval (Union[timedelta, int, None], default None):
If provided, use this as the size of the chunk boundaries.
Default to the value set in
pipe.parameters['verify']['chunk_minutes'](43200 — 30 days). - bounded (Optional[bool], default None):
If
True, do not verify older than the oldest sync time or newer than the newest. IfFalse, verify unbounded syncs outside of the new and old sync times. The default behavior (None) is to bound only if a bound interval is set (e.g.pipe.parameters['verify']['bound_days']). - deduplicate (bool, default False):
If
True, deduplicate the pipe's table after the verification syncs. - workers (Optional[int], default None):
If provided, limit the verification to this many threads.
Use a value of
1to sync chunks in series. - batchsize (Optional[int], default None):
If provided, sync this many chunks in parallel.
Defaults to
Pipe.get_num_workers(). - skip_chunks_with_greater_rowcounts (bool, default False):
If
True, compare the rowcounts for a chunk and skip syncing if the pipe's chunk rowcount equals or exceeds the remote's rowcount. - check_rowcounts_only (bool, default False):
If
True, only compare rowcounts and print chunks which are out-of-sync. - debug (bool, default False): Verbosity toggle.
- kwargs (Any):
All keyword arguments are passed to
pipe.sync().
Returns
- A SuccessTuple indicating whether the pipe was successfully resynced.
547def get_bound_interval(self, debug: bool = False) -> Union[timedelta, int, None]: 548 """ 549 Return the interval used to determine the bound time (limit for verification syncs). 550 If the datetime axis is an integer, just return its value. 551 552 Below are the supported keys for the bound interval: 553 554 - `pipe.parameters['verify']['bound_minutes']` 555 - `pipe.parameters['verify']['bound_hours']` 556 - `pipe.parameters['verify']['bound_days']` 557 - `pipe.parameters['verify']['bound_weeks']` 558 - `pipe.parameters['verify']['bound_years']` 559 - `pipe.parameters['verify']['bound_seconds']` 560 561 If multiple keys are present, the first on this priority list will be used. 562 563 Returns 564 ------- 565 A `timedelta` or `int` value to be used to determine the bound time. 566 """ 567 verify_params = self.parameters.get('verify', {}) 568 prefix = 'bound_' 569 suffixes_to_check = ('minutes', 'hours', 'days', 'weeks', 'years', 'seconds') 570 keys_to_search = { 571 key: val 572 for key, val in verify_params.items() 573 if key.startswith(prefix) 574 } 575 bound_time_key, bound_time_value = None, None 576 for key, value in keys_to_search.items(): 577 for suffix in suffixes_to_check: 578 if key == prefix + suffix: 579 bound_time_key = key 580 bound_time_value = value 581 break 582 if bound_time_key is not None: 583 break 584 585 if bound_time_value is None: 586 return bound_time_value 587 588 dt_col = self.columns.get('datetime', None) 589 if not dt_col: 590 return bound_time_value 591 592 dt_typ = self.dtypes.get(dt_col, 'datetime') 593 if 'int' in dt_typ.lower(): 594 return int(bound_time_value) 595 596 interval_type = bound_time_key.replace(prefix, '') 597 return timedelta(**{interval_type: bound_time_value})
Return the interval used to determine the bound time (limit for verification syncs). If the datetime axis is an integer, just return its value.
Below are the supported keys for the bound interval:
- `pipe.parameters['verify']['bound_minutes']`
- `pipe.parameters['verify']['bound_hours']`
- `pipe.parameters['verify']['bound_days']`
- `pipe.parameters['verify']['bound_weeks']`
- `pipe.parameters['verify']['bound_years']`
- `pipe.parameters['verify']['bound_seconds']`
If multiple keys are present, the first on this priority list will be used.
Returns
- A
timedeltaorintvalue to be used to determine the bound time.
600def get_bound_time(self, debug: bool = False) -> Union[datetime, int, None]: 601 """ 602 The bound time is the limit at which long-running verification syncs should stop. 603 A value of `None` means verification syncs should be unbounded. 604 605 Like deriving a backtrack time from `pipe.get_sync_time()`, 606 the bound time is the sync time minus a large window (e.g. 366 days). 607 608 Unbound verification syncs (i.e. `bound_time is None`) 609 if the oldest sync time is less than the bound interval. 610 611 Returns 612 ------- 613 A `datetime` or `int` corresponding to the 614 `begin` bound for verification and deduplication syncs. 615 """ 616 bound_interval = self.get_bound_interval(debug=debug) 617 if bound_interval is None: 618 return None 619 620 sync_time = self.get_sync_time(debug=debug) 621 if sync_time is None: 622 return None 623 624 bound_time = sync_time - bound_interval 625 oldest_sync_time = self.get_sync_time(newest=False, debug=debug) 626 max_bound_time_days = STATIC_CONFIG['pipes']['max_bound_time_days'] 627 628 extreme_sync_times_delta = ( 629 hasattr(oldest_sync_time, 'tzinfo') 630 and (sync_time - oldest_sync_time) >= timedelta(days=max_bound_time_days) 631 ) 632 633 return ( 634 bound_time 635 if bound_time > oldest_sync_time or extreme_sync_times_delta 636 else None 637 )
The bound time is the limit at which long-running verification syncs should stop.
A value of None means verification syncs should be unbounded.
Like deriving a backtrack time from pipe.get_sync_time(),
the bound time is the sync time minus a large window (e.g. 366 days).
Unbound verification syncs (i.e. bound_time is None)
if the oldest sync time is less than the bound interval.
Returns
- A
datetimeorintcorresponding to the beginbound for verification and deduplication syncs.
12def delete( 13 self, 14 drop: bool = True, 15 debug: bool = False, 16 **kw 17) -> SuccessTuple: 18 """ 19 Call the Pipe's instance connector's `delete_pipe()` method. 20 21 Parameters 22 ---------- 23 drop: bool, default True 24 If `True`, drop the pipes' target table. 25 26 debug : bool, default False 27 Verbosity toggle. 28 29 Returns 30 ------- 31 A `SuccessTuple` of success (`bool`), message (`str`). 32 33 """ 34 from meerschaum.utils.warnings import warn 35 from meerschaum.utils.venv import Venv 36 from meerschaum.connectors import get_connector_plugin 37 38 if self.temporary: 39 if self.cache: 40 invalidate_success, invalidate_msg = self._invalidate_cache(hard=True, debug=debug) 41 if not invalidate_success: 42 return invalidate_success, invalidate_msg 43 44 return ( 45 False, 46 "Cannot delete pipes created with `temporary=True` (read-only). " 47 + "You may want to call `pipe.drop()` instead." 48 ) 49 50 if drop: 51 drop_success, drop_msg = self.drop(debug=debug) 52 if not drop_success: 53 warn(f"Failed to drop {self}:\n{drop_msg}") 54 55 with Venv(get_connector_plugin(self.instance_connector)): 56 result = self.instance_connector.delete_pipe(self, debug=debug, **kw) 57 58 if not isinstance(result, tuple): 59 return False, f"Received an unexpected result from '{self.instance_connector}': {result}" 60 61 if result[0]: 62 self._invalidate_cache(hard=True, debug=debug) 63 self._clear_cache_key('_id', debug=debug) 64 65 return result
Call the Pipe's instance connector's delete_pipe() method.
Parameters
- drop (bool, default True):
If
True, drop the pipes' target table. - debug (bool, default False): Verbosity toggle.
Returns
- A
SuccessTupleof success (bool), message (str).
14def drop( 15 self, 16 debug: bool = False, 17 **kw: Any 18) -> SuccessTuple: 19 """ 20 Call the Pipe's instance connector's `drop_pipe()` method. 21 22 Parameters 23 ---------- 24 debug: bool, default False: 25 Verbosity toggle. 26 27 Returns 28 ------- 29 A `SuccessTuple` of success, message. 30 31 """ 32 from meerschaum.utils.venv import Venv 33 from meerschaum.connectors import get_connector_plugin 34 35 self._clear_cache_key('_exists', debug=debug) 36 37 with Venv(get_connector_plugin(self.instance_connector)): 38 if hasattr(self.instance_connector, 'drop_pipe'): 39 result = self.instance_connector.drop_pipe(self, debug=debug, **kw) 40 else: 41 result = ( 42 False, 43 ( 44 "Cannot drop pipes for instance connectors of type " 45 f"'{self.instance_connector.type}'." 46 ) 47 ) 48 49 self._clear_cache_key('_exists', debug=debug) 50 self._clear_cache_key('_exists_timestamp', debug=debug) 51 52 return result
Call the Pipe's instance connector's drop_pipe() method.
Parameters
- debug (bool, default False:): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
55def drop_indices( 56 self, 57 columns: Optional[List[str]] = None, 58 debug: bool = False, 59 **kw: Any 60) -> SuccessTuple: 61 """ 62 Call the Pipe's instance connector's `drop_indices()` method. 63 64 Parameters 65 ---------- 66 columns: Optional[List[str]] = None 67 If provided, only drop indices in the given list. 68 69 debug: bool, default False: 70 Verbosity toggle. 71 72 Returns 73 ------- 74 A `SuccessTuple` of success, message. 75 76 """ 77 from meerschaum.utils.venv import Venv 78 from meerschaum.connectors import get_connector_plugin 79 80 self._clear_cache_key('_columns_indices', debug=debug) 81 self._clear_cache_key('_columns_indices_timestamp', debug=debug) 82 self._clear_cache_key('_columns_types', debug=debug) 83 self._clear_cache_key('_columns_types_timestamp', debug=debug) 84 85 with Venv(get_connector_plugin(self.instance_connector)): 86 if hasattr(self.instance_connector, 'drop_pipe_indices'): 87 result = self.instance_connector.drop_pipe_indices( 88 self, 89 columns=columns, 90 debug=debug, 91 **kw 92 ) 93 else: 94 result = ( 95 False, 96 ( 97 "Cannot drop indices for instance connectors of type " 98 f"'{self.instance_connector.type}'." 99 ) 100 ) 101 102 self._clear_cache_key('_columns_indices', debug=debug) 103 self._clear_cache_key('_columns_indices_timestamp', debug=debug) 104 self._clear_cache_key('_columns_types', debug=debug) 105 self._clear_cache_key('_columns_types_timestamp', debug=debug) 106 107 return result
Call the Pipe's instance connector's drop_indices() method.
Parameters
- columns (Optional[List[str]] = None): If provided, only drop indices in the given list.
- debug (bool, default False:): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
14def compress( 15 self, 16 debug: bool = False, 17 **kw: Any 18) -> SuccessTuple: 19 """ 20 Call the Pipe's instance connector's `compress_pipe()` method. 21 22 For TimescaleDB hypertables this enables and applies native compression. 23 Other flavors fall back to their respective compression mechanisms where supported. 24 25 Parameters 26 ---------- 27 debug: bool, default False: 28 Verbosity toggle. 29 30 Returns 31 ------- 32 A `SuccessTuple` of success, message. 33 """ 34 from meerschaum.utils.venv import Venv 35 from meerschaum.connectors import get_connector_plugin 36 37 try: 38 with Venv(get_connector_plugin(self.instance_connector)): 39 if hasattr(self.instance_connector, 'compress_pipe'): 40 result = self.instance_connector.compress_pipe(self, debug=debug, **kw) 41 else: 42 result = ( 43 False, 44 ( 45 "Cannot compress pipes for instance connectors of type " 46 f"'{self.instance_connector.type}'." 47 ) 48 ) 49 except NotImplementedError: 50 result = ( 51 False, 52 ( 53 "Compression is not implemented for instance connectors of type " 54 f"'{self.instance_connector.type}'." 55 ) 56 ) 57 58 self._clear_cache_key('_exists', debug=debug) 59 return result
Call the Pipe's instance connector's compress_pipe() method.
For TimescaleDB hypertables this enables and applies native compression. Other flavors fall back to their respective compression mechanisms where supported.
Parameters
- debug (bool, default False:): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
62def decompress( 63 self, 64 debug: bool = False, 65 **kw: Any 66) -> SuccessTuple: 67 """ 68 Call the Pipe's instance connector's `decompress_pipe()` method, the inverse of `compress()`. 69 70 For TimescaleDB hypertables this removes the compression policy and converts compressed 71 chunks back to row-store. Other flavors fall back to their respective mechanisms where 72 supported. 73 74 Parameters 75 ---------- 76 debug: bool, default False: 77 Verbosity toggle. 78 79 Returns 80 ------- 81 A `SuccessTuple` of success, message. 82 """ 83 from meerschaum.utils.venv import Venv 84 from meerschaum.connectors import get_connector_plugin 85 86 try: 87 with Venv(get_connector_plugin(self.instance_connector)): 88 if hasattr(self.instance_connector, 'decompress_pipe'): 89 result = self.instance_connector.decompress_pipe(self, debug=debug, **kw) 90 else: 91 result = ( 92 False, 93 ( 94 "Cannot decompress pipes for instance connectors of type " 95 f"'{self.instance_connector.type}'." 96 ) 97 ) 98 except NotImplementedError: 99 result = ( 100 False, 101 ( 102 "Decompression is not implemented for instance connectors of type " 103 f"'{self.instance_connector.type}'." 104 ) 105 ) 106 107 self._clear_cache_key('_exists', debug=debug) 108 return result
Call the Pipe's instance connector's decompress_pipe() method, the inverse of compress().
For TimescaleDB hypertables this removes the compression policy and converts compressed chunks back to row-store. Other flavors fall back to their respective mechanisms where supported.
Parameters
- debug (bool, default False:): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
14def vacuum( 15 self, 16 full: bool = False, 17 debug: bool = False, 18 **kw: Any 19) -> SuccessTuple: 20 """ 21 Call the Pipe's instance connector's `vacuum_pipe()` method to reclaim disk space. 22 23 For PostgreSQL-family tables this runs `VACUUM` (optionally `VACUUM FULL`); other flavors 24 fall back to their respective space-reclaiming mechanisms where supported. 25 26 Parameters 27 ---------- 28 full: bool, default False 29 If `True` (PostgreSQL family only), run `VACUUM FULL` to return freed space to the OS. 30 31 debug: bool, default False 32 Verbosity toggle. 33 34 Returns 35 ------- 36 A `SuccessTuple` of success, message. 37 """ 38 from meerschaum.utils.venv import Venv 39 from meerschaum.connectors import get_connector_plugin 40 41 try: 42 with Venv(get_connector_plugin(self.instance_connector)): 43 if hasattr(self.instance_connector, 'vacuum_pipe'): 44 result = self.instance_connector.vacuum_pipe(self, full=full, debug=debug, **kw) 45 else: 46 result = ( 47 False, 48 ( 49 "Cannot vacuum pipes for instance connectors of type " 50 f"'{self.instance_connector.type}'." 51 ) 52 ) 53 except NotImplementedError: 54 result = ( 55 False, 56 ( 57 "Vacuuming is not implemented for instance connectors of type " 58 f"'{self.instance_connector.type}'." 59 ) 60 ) 61 62 self._clear_cache_key('_exists', debug=debug) 63 return result
Call the Pipe's instance connector's vacuum_pipe() method to reclaim disk space.
For PostgreSQL-family tables this runs VACUUM (optionally VACUUM FULL); other flavors
fall back to their respective space-reclaiming mechanisms where supported.
Parameters
- full (bool, default False):
If
True(PostgreSQL family only), runVACUUM FULLto return freed space to the OS. - debug (bool, default False): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
66def analyze( 67 self, 68 debug: bool = False, 69 **kw: Any 70) -> SuccessTuple: 71 """ 72 Call the Pipe's instance connector's `analyze_pipe()` method to refresh planner statistics. 73 74 Parameters 75 ---------- 76 debug: bool, default False 77 Verbosity toggle. 78 79 Returns 80 ------- 81 A `SuccessTuple` of success, message. 82 """ 83 from meerschaum.utils.venv import Venv 84 from meerschaum.connectors import get_connector_plugin 85 86 try: 87 with Venv(get_connector_plugin(self.instance_connector)): 88 if hasattr(self.instance_connector, 'analyze_pipe'): 89 result = self.instance_connector.analyze_pipe(self, debug=debug, **kw) 90 else: 91 result = ( 92 False, 93 ( 94 "Cannot analyze pipes for instance connectors of type " 95 f"'{self.instance_connector.type}'." 96 ) 97 ) 98 except NotImplementedError: 99 result = ( 100 False, 101 ( 102 "Analyzing is not implemented for instance connectors of type " 103 f"'{self.instance_connector.type}'." 104 ) 105 ) 106 107 return result
Call the Pipe's instance connector's analyze_pipe() method to refresh planner statistics.
Parameters
- debug (bool, default False): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
110def repartition( 111 self, 112 chunk_minutes: Optional[int] = None, 113 debug: bool = False, 114 **kw: Any 115) -> SuccessTuple: 116 """ 117 Call the Pipe's instance connector's `partition_pipe()` method to rebuild the target table 118 to a new partition (chunk) width. 119 120 On TimescaleDB this changes the chunk interval for future chunks. On PostgreSQL / PostGIS, 121 MySQL / MariaDB, and MSSQL it rebuilds the natively range-partitioned table at the new width. 122 123 Parameters 124 ---------- 125 chunk_minutes: Optional[int], default None 126 The new partition width in minutes. Defaults to the pipe's `verify.chunk_minutes`. 127 128 debug: bool, default False 129 Verbosity toggle. 130 131 Returns 132 ------- 133 A `SuccessTuple` of success, message. 134 """ 135 from meerschaum.utils.venv import Venv 136 from meerschaum.connectors import get_connector_plugin 137 138 try: 139 with Venv(get_connector_plugin(self.instance_connector)): 140 if hasattr(self.instance_connector, 'partition_pipe'): 141 result = self.instance_connector.partition_pipe( 142 self, chunk_minutes=chunk_minutes, debug=debug, **kw 143 ) 144 else: 145 result = ( 146 False, 147 ( 148 "Cannot repartition pipes for instance connectors of type " 149 f"'{self.instance_connector.type}'." 150 ) 151 ) 152 except NotImplementedError: 153 result = ( 154 False, 155 ( 156 "Repartitioning is not implemented for instance connectors of type " 157 f"'{self.instance_connector.type}'." 158 ) 159 ) 160 161 self._clear_cache_key('_exists', debug=debug) 162 return result
Call the Pipe's instance connector's partition_pipe() method to rebuild the target table
to a new partition (chunk) width.
On TimescaleDB this changes the chunk interval for future chunks. On PostgreSQL / PostGIS, MySQL / MariaDB, and MSSQL it rebuilds the natively range-partitioned table at the new width.
Parameters
- chunk_minutes (Optional[int], default None):
The new partition width in minutes. Defaults to the pipe's
verify.chunk_minutes. - debug (bool, default False): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
14def create_indices( 15 self, 16 columns: Optional[List[str]] = None, 17 debug: bool = False, 18 **kw: Any 19) -> SuccessTuple: 20 """ 21 Call the Pipe's instance connector's `create_pipe_indices()` method. 22 23 Parameters 24 ---------- 25 debug: bool, default False: 26 Verbosity toggle. 27 28 Returns 29 ------- 30 A `SuccessTuple` of success, message. 31 32 """ 33 from meerschaum.utils.venv import Venv 34 from meerschaum.connectors import get_connector_plugin 35 36 self._clear_cache_key('_columns_indices', debug=debug) 37 self._clear_cache_key('_columns_indices_timestamp', debug=debug) 38 self._clear_cache_key('_columns_types', debug=debug) 39 self._clear_cache_key('_columns_types_timestamp', debug=debug) 40 41 with Venv(get_connector_plugin(self.instance_connector)): 42 if hasattr(self.instance_connector, 'create_pipe_indices'): 43 result = self.instance_connector.create_pipe_indices( 44 self, 45 columns=columns, 46 debug=debug, 47 **kw 48 ) 49 else: 50 result = ( 51 False, 52 ( 53 "Cannot create indices for instance connectors of type " 54 f"'{self.instance_connector.type}'." 55 ) 56 ) 57 58 self._clear_cache_key('_columns_indices', debug=debug) 59 self._clear_cache_key('_columns_indices_timestamp', debug=debug) 60 self._clear_cache_key('_columns_types', debug=debug) 61 self._clear_cache_key('_columns_types_timestamp', debug=debug) 62 63 return result
Call the Pipe's instance connector's create_pipe_indices() method.
Parameters
- debug (bool, default False:): Verbosity toggle.
Returns
- A
SuccessTupleof success, message.
16def clear( 17 self, 18 begin: Optional[datetime] = None, 19 end: Optional[datetime] = None, 20 params: Optional[Dict[str, Any]] = None, 21 debug: bool = False, 22 **kwargs: Any 23) -> SuccessTuple: 24 """ 25 Call the Pipe's instance connector's `clear_pipe` method. 26 27 Parameters 28 ---------- 29 begin: Optional[datetime], default None: 30 If provided, only remove rows newer than this datetime value. 31 32 end: Optional[datetime], default None: 33 If provided, only remove rows older than this datetime column (not including end). 34 35 params: Optional[Dict[str, Any]], default None 36 See `meerschaum.utils.sql.build_where`. 37 38 debug: bool, default False: 39 Verbositity toggle. 40 41 Returns 42 ------- 43 A `SuccessTuple` corresponding to whether this procedure completed successfully. 44 45 Examples 46 -------- 47 >>> pipe = mrsm.Pipe('test', 'test', columns={'datetime': 'dt'}, instance='sql:local') 48 >>> pipe.sync({'dt': [datetime(2020, 1, 1, 0, 0)]}) 49 >>> pipe.sync({'dt': [datetime(2021, 1, 1, 0, 0)]}) 50 >>> pipe.sync({'dt': [datetime(2022, 1, 1, 0, 0)]}) 51 >>> 52 >>> pipe.clear(begin=datetime(2021, 1, 1, 0, 0)) 53 >>> pipe.get_data() 54 dt 55 0 2020-01-01 56 57 """ 58 from meerschaum.utils.warnings import warn 59 from meerschaum.utils.venv import Venv 60 from meerschaum.connectors import get_connector_plugin 61 62 begin, end = self.parse_date_bounds(begin, end) 63 64 with Venv(get_connector_plugin(self.instance_connector)): 65 return self.instance_connector.clear_pipe( 66 self, 67 begin=begin, 68 end=end, 69 params=params, 70 debug=debug, 71 **kwargs 72 )
Call the Pipe's instance connector's clear_pipe method.
Parameters
- begin (Optional[datetime], default None:): If provided, only remove rows newer than this datetime value.
- end (Optional[datetime], default None:): If provided, only remove rows older than this datetime column (not including end).
- params (Optional[Dict[str, Any]], default None):
See
meerschaum.utils.sql.build_where. - debug (bool, default False:): Verbositity toggle.
Returns
- A
SuccessTuplecorresponding to whether this procedure completed successfully.
Examples
>>> pipe = mrsm.Pipe('test', 'test', columns={'datetime': 'dt'}, instance='sql:local')
>>> pipe.sync({'dt': [datetime(2020, 1, 1, 0, 0)]})
>>> pipe.sync({'dt': [datetime(2021, 1, 1, 0, 0)]})
>>> pipe.sync({'dt': [datetime(2022, 1, 1, 0, 0)]})
>>>
>>> pipe.clear(begin=datetime(2021, 1, 1, 0, 0))
>>> pipe.get_data()
dt
0 2020-01-01
15def deduplicate( 16 self, 17 begin: Union[datetime, int, None] = None, 18 end: Union[datetime, int, None] = None, 19 params: Optional[Dict[str, Any]] = None, 20 chunk_interval: Union[datetime, int, None] = None, 21 bounded: Optional[bool] = None, 22 workers: Optional[int] = None, 23 debug: bool = False, 24 _use_instance_method: bool = True, 25 **kwargs: Any 26) -> SuccessTuple: 27 """ 28 Call the Pipe's instance connector's `delete_duplicates` method to delete duplicate rows. 29 30 Parameters 31 ---------- 32 begin: Union[datetime, int, None], default None: 33 If provided, only deduplicate rows newer than this datetime value. 34 35 end: Union[datetime, int, None], default None: 36 If provided, only deduplicate rows older than this datetime column (not including end). 37 38 params: Optional[Dict[str, Any]], default None 39 Restrict deduplication to this filter (for multiplexed data streams). 40 See `meerschaum.utils.sql.build_where`. 41 42 chunk_interval: Union[timedelta, int, None], default None 43 If provided, use this for the chunk bounds. 44 Defaults to the value set in `pipe.parameters['verify']['chunk_minutes']` (43200 — 30 days). 45 46 bounded: Optional[bool], default None 47 Only check outside the oldest and newest sync times if bounded is explicitly `False`. 48 49 workers: Optional[int], default None 50 If the instance connector is thread-safe, limit concurrenct syncs to this many threads. 51 52 debug: bool, default False: 53 Verbositity toggle. 54 55 kwargs: Any 56 All other keyword arguments are passed to 57 `pipe.sync()`, `pipe.clear()`, and `pipe.get_data(). 58 59 Returns 60 ------- 61 A `SuccessTuple` corresponding to whether all of the chunks were successfully deduplicated. 62 """ 63 from meerschaum.utils.warnings import warn, info 64 from meerschaum.utils.misc import interval_str, items_str 65 from meerschaum.utils.venv import Venv 66 from meerschaum.connectors import get_connector_plugin 67 from meerschaum.utils.pool import get_pool 68 69 begin, end = self.parse_date_bounds(begin, end) 70 71 workers = self.get_num_workers(workers=workers) 72 pool = get_pool(workers=workers) 73 74 if _use_instance_method: 75 with Venv(get_connector_plugin(self.instance_connector)): 76 if hasattr(self.instance_connector, 'deduplicate_pipe'): 77 return self.instance_connector.deduplicate_pipe( 78 self, 79 begin=begin, 80 end=end, 81 params=params, 82 bounded=bounded, 83 debug=debug, 84 **kwargs 85 ) 86 87 ### Only unbound if explicitly False. 88 if bounded is None: 89 bounded = True 90 chunk_interval = self.get_chunk_interval(chunk_interval, debug=debug) 91 92 bound_time = self.get_bound_time(debug=debug) 93 if bounded and begin is None: 94 begin = ( 95 bound_time 96 if bound_time is not None 97 else self.get_sync_time(newest=False, debug=debug) 98 ) 99 if bounded and end is None: 100 end = self.get_sync_time(newest=True, debug=debug) 101 if end is not None: 102 end += ( 103 timedelta(minutes=1) 104 if hasattr(end, 'tzinfo') 105 else 1 106 ) 107 108 chunk_bounds = self.get_chunk_bounds( 109 bounded=bounded, 110 begin=begin, 111 end=end, 112 chunk_interval=chunk_interval, 113 debug=debug, 114 ) 115 116 indices = [col for col in self.columns.values() if col] 117 if not indices: 118 return False, "Cannot deduplicate without index columns." 119 120 def process_chunk_bounds(bounds) -> Tuple[ 121 Tuple[ 122 Union[datetime, int, None], 123 Union[datetime, int, None] 124 ], 125 SuccessTuple 126 ]: 127 ### Only selecting the index values here to keep bandwidth down. 128 chunk_begin, chunk_end = bounds 129 chunk_df = self.get_data( 130 select_columns=indices, 131 begin=chunk_begin, 132 end=chunk_end, 133 params=params, 134 debug=debug, 135 ) 136 if chunk_df is None: 137 return bounds, (True, "") 138 existing_chunk_len = len(chunk_df) 139 deduped_chunk_df = chunk_df.drop_duplicates(keep='last') 140 deduped_chunk_len = len(deduped_chunk_df) 141 142 if existing_chunk_len == deduped_chunk_len: 143 return bounds, (True, "") 144 145 chunk_msg_header = f"\n{chunk_begin} - {chunk_end}" 146 chunk_msg_body = "" 147 148 full_chunk = self.get_data( 149 begin=chunk_begin, 150 end=chunk_end, 151 params=params, 152 debug=debug, 153 ) 154 if full_chunk is None or len(full_chunk) == 0: 155 return bounds, (True, f"{chunk_msg_header}\nChunk is empty, skipping...") 156 157 chunk_indices = [ix for ix in indices if ix in full_chunk.columns] 158 if not chunk_indices: 159 return bounds, (False, f"None of {items_str(indices)} were present in chunk.") 160 try: 161 full_chunk = full_chunk.drop_duplicates( 162 subset=chunk_indices, 163 keep='last' 164 ).reset_index( 165 drop=True, 166 ) 167 except Exception as e: 168 return ( 169 bounds, 170 (False, f"Failed to deduplicate chunk on {items_str(chunk_indices)}:\n({e})") 171 ) 172 173 clear_success, clear_msg = self.clear( 174 begin=chunk_begin, 175 end=chunk_end, 176 params=params, 177 debug=debug, 178 ) 179 if not clear_success: 180 chunk_msg_body += f"Failed to clear chunk while deduplicating:\n{clear_msg}\n" 181 warn(chunk_msg_body) 182 183 sync_success, sync_msg = self.sync(full_chunk, debug=debug) 184 if not sync_success: 185 chunk_msg_body += f"Failed to sync chunk while deduplicating:\n{sync_msg}\n" 186 187 ### Finally check if the deduplication worked. 188 chunk_rowcount = self.get_rowcount( 189 begin=chunk_begin, 190 end=chunk_end, 191 params=params, 192 debug=debug, 193 ) 194 if chunk_rowcount != deduped_chunk_len: 195 return bounds, ( 196 False, ( 197 chunk_msg_header + "\n" 198 + chunk_msg_body + ("\n" if chunk_msg_body else '') 199 + "Chunk rowcounts still differ (" 200 + f"{chunk_rowcount} rowcount vs {deduped_chunk_len} chunk length)." 201 ) 202 ) 203 204 return bounds, ( 205 True, ( 206 chunk_msg_header + "\n" 207 + chunk_msg_body + ("\n" if chunk_msg_body else '') 208 + f"Deduplicated chunk from {existing_chunk_len} to {chunk_rowcount} rows." 209 ) 210 ) 211 212 info( 213 f"Deduplicating {len(chunk_bounds)} chunk" 214 + ('s' if len(chunk_bounds) != 1 else '') 215 + f" ({'un' if not bounded else ''}bounded)" 216 + f" of size '{interval_str(chunk_interval)}'" 217 + f" on {self}." 218 ) 219 bounds_success_tuples = dict(pool.map(process_chunk_bounds, chunk_bounds)) 220 bounds_successes = { 221 bounds: success_tuple 222 for bounds, success_tuple in bounds_success_tuples.items() 223 if success_tuple[0] 224 } 225 bounds_failures = { 226 bounds: success_tuple 227 for bounds, success_tuple in bounds_success_tuples.items() 228 if not success_tuple[0] 229 } 230 231 ### No need to retry if everything failed. 232 if len(bounds_failures) > 0 and len(bounds_successes) == 0: 233 return ( 234 False, 235 ( 236 f"Failed to deduplicate {len(bounds_failures)} chunk" 237 + ('s' if len(bounds_failures) != 1 else '') 238 + ".\n" 239 + "\n".join([msg for _, (_, msg) in bounds_failures.items() if msg]) 240 ) 241 ) 242 243 retry_bounds = [bounds for bounds in bounds_failures] 244 if not retry_bounds: 245 return ( 246 True, 247 ( 248 f"Successfully deduplicated {len(bounds_successes)} chunk" 249 + ('s' if len(bounds_successes) != 1 else '') 250 + ".\n" 251 + "\n".join([msg for _, (_, msg) in bounds_successes.items() if msg]) 252 ).rstrip('\n') 253 ) 254 255 info(f"Retrying {len(retry_bounds)} chunks for {self}...") 256 retry_bounds_success_tuples = dict(pool.map(process_chunk_bounds, retry_bounds)) 257 retry_bounds_successes = { 258 bounds: success_tuple 259 for bounds, success_tuple in retry_bounds_success_tuples.items() 260 if success_tuple[0] 261 } 262 retry_bounds_failures = { 263 bounds: success_tuple 264 for bounds, success_tuple in retry_bounds_success_tuples.items() 265 if not success_tuple[0] 266 } 267 268 bounds_successes.update(retry_bounds_successes) 269 if not retry_bounds_failures: 270 return ( 271 True, 272 ( 273 f"Successfully deduplicated {len(bounds_successes)} chunk" 274 + ('s' if len(bounds_successes) != 1 else '') 275 + f"({len(retry_bounds_successes)} retried):\n" 276 + "\n".join([msg for _, (_, msg) in bounds_successes.items() if msg]) 277 ).rstrip('\n') 278 ) 279 280 return ( 281 False, 282 ( 283 f"Failed to deduplicate {len(bounds_failures)} chunk" 284 + ('s' if len(retry_bounds_failures) != 1 else '') 285 + ".\n" 286 + "\n".join([msg for _, (_, msg) in retry_bounds_failures.items() if msg]) 287 ).rstrip('\n') 288 )
Call the Pipe's instance connector's delete_duplicates method to delete duplicate rows.
Parameters
- begin (Union[datetime, int, None], default None:): If provided, only deduplicate rows newer than this datetime value.
- end (Union[datetime, int, None], default None:): If provided, only deduplicate rows older than this datetime column (not including end).
- params (Optional[Dict[str, Any]], default None):
Restrict deduplication to this filter (for multiplexed data streams).
See
meerschaum.utils.sql.build_where. - chunk_interval (Union[timedelta, int, None], default None):
If provided, use this for the chunk bounds.
Defaults to the value set in
pipe.parameters['verify']['chunk_minutes'](43200 — 30 days). - bounded (Optional[bool], default None):
Only check outside the oldest and newest sync times if bounded is explicitly
False. - workers (Optional[int], default None): If the instance connector is thread-safe, limit concurrenct syncs to this many threads.
- debug (bool, default False:): Verbositity toggle.
- kwargs (Any):
All other keyword arguments are passed to
pipe.sync(),pipe.clear(), and `pipe.get_data().
Returns
- A
SuccessTuplecorresponding to whether all of the chunks were successfully deduplicated.
16def bootstrap( 17 self, 18 debug: bool = False, 19 yes: bool = False, 20 force: bool = False, 21 noask: bool = False, 22 shell: bool = False, 23 **kw 24) -> SuccessTuple: 25 """ 26 Prompt the user to create a pipe's requirements all from one method. 27 This method shouldn't be used in any automated scripts because it interactively 28 prompts the user and therefore may hang. 29 30 Parameters 31 ---------- 32 debug: bool, default False: 33 Verbosity toggle. 34 35 yes: bool, default False: 36 Print the questions and automatically agree. 37 38 force: bool, default False: 39 Skip the questions and agree anyway. 40 41 noask: bool, default False: 42 Print the questions but go with the default answer. 43 44 shell: bool, default False: 45 Used to determine if we are in the interactive shell. 46 47 Returns 48 ------- 49 A `SuccessTuple` corresponding to the success of this procedure. 50 51 """ 52 53 from meerschaum.utils.warnings import info 54 from meerschaum.utils.prompt import prompt, yes_no 55 from meerschaum.utils.formatting import pprint 56 from meerschaum.config import get_config 57 from meerschaum.utils.formatting._shell import clear_screen 58 from meerschaum.utils.formatting import print_tuple 59 from meerschaum.actions import actions 60 from meerschaum.utils.venv import Venv 61 from meerschaum.connectors import get_connector_plugin 62 63 _clear = get_config('shell', 'clear_screen', patch=True) 64 65 if self.id is not None: 66 delete_tuple = self.delete(debug=debug) 67 if not delete_tuple[0]: 68 return delete_tuple 69 70 if _clear: 71 clear_screen(debug=debug) 72 73 _parameters = _get_parameters(self, debug=debug) 74 self.parameters = _parameters 75 pprint(self.parameters) 76 try: 77 prompt( 78 f"\n Press [Enter] to register {self} with the above configuration:", 79 icon = False 80 ) 81 except KeyboardInterrupt: 82 return False, f"Aborted bootstrapping {self}." 83 84 with Venv(get_connector_plugin(self.instance_connector)): 85 register_tuple = self.instance_connector.register_pipe(self, debug=debug) 86 87 if not register_tuple[0]: 88 return register_tuple 89 90 if _clear: 91 clear_screen(debug=debug) 92 93 try: 94 if yes_no( 95 f"Would you like to edit the definition for {self}?", 96 yes=yes, 97 noask=noask, 98 default='n', 99 ): 100 edit_tuple = self.edit_definition(debug=debug) 101 if not edit_tuple[0]: 102 return edit_tuple 103 104 if yes_no( 105 f"Would you like to try syncing {self} now?", 106 yes=yes, 107 noask=noask, 108 default='n', 109 ): 110 sync_tuple = actions['sync']( 111 ['pipes'], 112 connector_keys=[self.connector_keys], 113 metric_keys=[self.metric_key], 114 location_keys=[self.location_key], 115 mrsm_instance=str(self.instance_connector), 116 debug=debug, 117 shell=shell, 118 ) 119 if not sync_tuple[0]: 120 return sync_tuple 121 except Exception as e: 122 return False, f"Failed to bootstrap {self}:\n" + str(e) 123 124 print_tuple((True, f"Finished bootstrapping {self}!")) 125 info( 126 "You can edit this pipe later with `edit pipes` " 127 + "or set the definition with `edit pipes definition`.\n" 128 + " To sync data into your pipe, run `sync pipes`." 129 ) 130 131 return True, "Success"
Prompt the user to create a pipe's requirements all from one method. This method shouldn't be used in any automated scripts because it interactively prompts the user and therefore may hang.
Parameters
- debug (bool, default False:): Verbosity toggle.
- yes (bool, default False:): Print the questions and automatically agree.
- force (bool, default False:): Skip the questions and agree anyway.
- noask (bool, default False:): Print the questions but go with the default answer.
- shell (bool, default False:): Used to determine if we are in the interactive shell.
Returns
- A
SuccessTuplecorresponding to the success of this procedure.
20def enforce_dtypes( 21 self, 22 df: 'pd.DataFrame', 23 chunksize: Optional[int] = -1, 24 enforce: bool = True, 25 safe_copy: bool = True, 26 dtypes: Optional[Dict[str, str]] = None, 27 as_polars: bool = False, 28 debug: bool = False, 29) -> 'pd.DataFrame': 30 """ 31 Cast the input dataframe to the pipe's registered data types. 32 If the pipe does not exist and dtypes are not set, return the dataframe. 33 """ 34 import traceback 35 from meerschaum.utils.warnings import warn 36 from meerschaum.utils.debug import dprint 37 from meerschaum.utils.dataframe import ( 38 parse_df_datetimes, 39 enforce_dtypes as _enforce_dtypes, 40 parse_simple_lines, 41 to_pandas, 42 to_polars, 43 ) 44 from meerschaum.utils.dtypes import are_dtypes_equal 45 from meerschaum.utils.packages import import_pandas 46 pd = import_pandas(debug=debug) 47 if df is None: 48 if debug: 49 dprint( 50 "Received None instead of a DataFrame.\n" 51 + " Skipping dtype enforcement..." 52 ) 53 return df 54 55 if not self.enforce: 56 enforce = False 57 58 explicit_dtypes = self.get_dtypes(infer=False, debug=debug) if enforce else {} 59 pipe_dtypes = self.get_dtypes(infer=True, debug=debug) if not dtypes else dtypes 60 61 try: 62 if isinstance(df, str): 63 if df.strip() and df.strip()[0] not in ('{', '['): 64 df = parse_df_datetimes( 65 parse_simple_lines(df), 66 ignore_cols=[ 67 col 68 for col, dtype in pipe_dtypes.items() 69 if (not enforce or not are_dtypes_equal(dtype, 'datetime')) 70 ], 71 ) 72 else: 73 df = parse_df_datetimes( 74 pd.read_json(StringIO(df)), 75 ignore_cols=[ 76 col 77 for col, dtype in pipe_dtypes.items() 78 if (not enforce or not are_dtypes_equal(dtype, 'datetime')) 79 ], 80 ignore_all=(not enforce), 81 strip_timezone=(self.tzinfo is None), 82 chunksize=chunksize, 83 debug=debug, 84 ) 85 elif isinstance(df, (dict, list, tuple)): 86 df = parse_df_datetimes( 87 df, 88 ignore_cols=[ 89 col 90 for col, dtype in pipe_dtypes.items() 91 if (not enforce or not are_dtypes_equal(str(dtype), 'datetime')) 92 ], 93 strip_timezone=(self.tzinfo is None), 94 chunksize=chunksize, 95 debug=debug, 96 ) 97 except Exception as e: 98 warn(f"Unable to cast incoming data as a DataFrame...:\n{e}\n\n{traceback.format_exc()}") 99 return None 100 101 if not pipe_dtypes: 102 if debug: 103 dprint( 104 f"Could not find dtypes for {self}.\n" 105 + "Skipping dtype enforcement..." 106 ) 107 return to_polars(df) if as_polars else to_pandas(df) 108 109 return _enforce_dtypes( 110 df, 111 pipe_dtypes, 112 explicit_dtypes=explicit_dtypes, 113 safe_copy=safe_copy, 114 strip_timezone=(self.tzinfo is None), 115 coerce_numeric=self.mixed_numerics, 116 coerce_timezone=enforce, 117 as_polars=as_polars, 118 debug=debug, 119 )
Cast the input dataframe to the pipe's registered data types. If the pipe does not exist and dtypes are not set, return the dataframe.
122def infer_dtypes( 123 self, 124 persist: bool = False, 125 refresh: bool = False, 126 debug: bool = False, 127) -> Dict[str, Any]: 128 """ 129 If `dtypes` is not set in `meerschaum.Pipe.parameters`, 130 infer the data types from the underlying table if it exists. 131 132 Parameters 133 ---------- 134 persist: bool, default False 135 If `True`, persist the inferred data types to `meerschaum.Pipe.parameters`. 136 NOTE: Use with caution! Generally `dtypes` is meant to be user-configurable only. 137 138 refresh: bool, default False 139 If `True`, retrieve the latest columns-types for the pipe. 140 See `Pipe.get_columns.types()`. 141 142 Returns 143 ------- 144 A dictionary of strings containing the pandas data types for this Pipe. 145 """ 146 if not self.exists(debug=debug): 147 return {} 148 149 from meerschaum.utils.dtypes.sql import get_pd_type_from_db_type 150 from meerschaum.utils.dtypes import to_pandas_dtype 151 152 ### NOTE: get_columns_types() may return either the types as 153 ### PostgreSQL- or Pandas-style. 154 columns_types = self.get_columns_types(refresh=refresh, debug=debug) 155 156 remote_pd_dtypes = { 157 c: ( 158 get_pd_type_from_db_type(t, allow_custom_dtypes=True) 159 if str(t).isupper() 160 else to_pandas_dtype(t) 161 ) 162 for c, t in columns_types.items() 163 } if columns_types else {} 164 if not persist: 165 return remote_pd_dtypes 166 167 parameters = self.get_parameters(refresh=refresh, debug=debug) 168 dtypes = parameters.get('dtypes', {}) 169 dtypes.update({ 170 col: typ 171 for col, typ in remote_pd_dtypes.items() 172 if col not in dtypes 173 }) 174 self.dtypes = dtypes 175 self.edit(interactive=False, debug=debug) 176 return remote_pd_dtypes
If dtypes is not set in meerschaum.Pipe.parameters,
infer the data types from the underlying table if it exists.
Parameters
- persist (bool, default False):
If
True, persist the inferred data types tomeerschaum.Pipe.parameters. NOTE: Use with caution! Generallydtypesis meant to be user-configurable only. - refresh (bool, default False):
If
True, retrieve the latest columns-types for the pipe. SeePipe.get_columns.types().
Returns
- A dictionary of strings containing the pandas data types for this Pipe.
15def copy_to( 16 self, 17 instance_keys: str, 18 sync: bool = True, 19 begin: Union[datetime, int, None] = None, 20 end: Union[datetime, int, None] = None, 21 params: Optional[Dict[str, Any]] = None, 22 chunk_interval: Union[timedelta, int, None] = None, 23 debug: bool = False, 24 **kwargs: Any 25) -> SuccessTuple: 26 """ 27 Copy a pipe to another instance. 28 29 Parameters 30 ---------- 31 instance_keys: str 32 The instance to which to copy this pipe. 33 34 sync: bool, default True 35 If `True`, sync the source pipe's documents 36 37 begin: Union[datetime, int, None], default None 38 Beginning datetime value to pass to `Pipe.get_data()`. 39 40 end: Union[datetime, int, None], default None 41 End datetime value to pass to `Pipe.get_data()`. 42 43 params: Optional[Dict[str, Any]], default None 44 Parameters filter to pass to `Pipe.get_data()`. 45 46 chunk_interval: Union[timedelta, int, None], default None 47 The size of chunks to retrieve from `Pipe.get_data()` for syncing. 48 49 kwargs: Any 50 Additional flags to pass to `Pipe.get_data()` and `Pipe.sync()`, e.g. `workers`. 51 52 Returns 53 ------- 54 A SuccessTuple indicating success. 55 """ 56 if str(instance_keys) == self.instance_keys: 57 return False, f"Cannot copy {self} to instance '{instance_keys}'." 58 59 begin, end = self.parse_date_bounds(begin, end) 60 61 new_pipe = mrsm.Pipe( 62 self.connector_keys, 63 self.metric_key, 64 self.location_key, 65 parameters=self.parameters.copy(), 66 instance=instance_keys, 67 ) 68 69 new_pipe_is_registered = new_pipe.id is not None 70 71 metadata_method = new_pipe.edit if new_pipe_is_registered else new_pipe.register 72 metadata_success, metadata_msg = metadata_method(debug=debug) 73 if not metadata_success: 74 return metadata_success, metadata_msg 75 76 if not self.exists(debug=debug): 77 return True, f"{self} does not exist; nothing to sync." 78 79 original_as_iterator = kwargs.get('as_iterator', None) 80 kwargs['as_iterator'] = True 81 82 chunk_generator = self.get_data( 83 begin=begin, 84 end=end, 85 params=params, 86 chunk_interval=chunk_interval, 87 debug=debug, 88 **kwargs 89 ) 90 91 if original_as_iterator is None: 92 _ = kwargs.pop('as_iterator', None) 93 else: 94 kwargs['as_iterator'] = original_as_iterator 95 96 sync_success, sync_msg = new_pipe.sync( 97 chunk_generator, 98 begin=begin, 99 end=end, 100 params=params, 101 debug=debug, 102 **kwargs 103 ) 104 msg = ( 105 f"Successfully synced {new_pipe}:\n{sync_msg}" 106 if sync_success 107 else f"Failed to sync {new_pipe}:\n{sync_msg}" 108 ) 109 return sync_success, msg
Copy a pipe to another instance.
Parameters
- instance_keys (str): The instance to which to copy this pipe.
- sync (bool, default True):
If
True, sync the source pipe's documents - begin (Union[datetime, int, None], default None):
Beginning datetime value to pass to
Pipe.get_data(). - end (Union[datetime, int, None], default None):
End datetime value to pass to
Pipe.get_data(). - params (Optional[Dict[str, Any]], default None):
Parameters filter to pass to
Pipe.get_data(). - chunk_interval (Union[timedelta, int, None], default None):
The size of chunks to retrieve from
Pipe.get_data()for syncing. - kwargs (Any):
Additional flags to pass to
Pipe.get_data()andPipe.sync(), e.g.workers.
Returns
- A SuccessTuple indicating success.
50class Plugin: 51 """Handle packaging of Meerschaum plugins.""" 52 53 def __init__( 54 self, 55 name: str, 56 version: Optional[str] = None, 57 user_id: Optional[int] = None, 58 required: Optional[List[str]] = None, 59 attributes: Optional[Dict[str, Any]] = None, 60 archive_path: Optional[pathlib.Path] = None, 61 venv_path: Optional[pathlib.Path] = None, 62 repo_connector: Optional['mrsm.connectors.api.APIConnector'] = None, 63 repo: Union['mrsm.connectors.api.APIConnector', str, None] = None, 64 ): 65 import meerschaum.config.paths as paths 66 from meerschaum._internal.static import STATIC_CONFIG 67 sep = STATIC_CONFIG['plugins']['repo_separator'] 68 _repo = None 69 if sep in name: 70 try: 71 name, _repo = name.split(sep) 72 except Exception as e: 73 error(f"Invalid plugin name: '{name}'") 74 self._repo_in_name = _repo 75 76 if attributes is None: 77 attributes = {} 78 self.name = name 79 self.attributes = attributes 80 self.user_id = user_id 81 self._version = version 82 if required: 83 self._required = required 84 self.archive_path = ( 85 archive_path if archive_path is not None 86 else paths.PLUGINS_ARCHIVES_RESOURCES_PATH / f"{self.name}.tar.gz" 87 ) 88 self.venv_path = ( 89 venv_path if venv_path is not None 90 else paths.VIRTENV_RESOURCES_PATH / self.name 91 ) 92 self._repo_connector = repo_connector 93 self._repo_keys = repo 94 95 96 @property 97 def repo_connector(self): 98 """ 99 Return the repository connector for this plugin. 100 NOTE: This imports the `connectors` module, which imports certain plugin modules. 101 """ 102 if self._repo_connector is None: 103 from meerschaum.connectors.parse import parse_repo_keys 104 105 repo_keys = self._repo_keys or self._repo_in_name 106 if self._repo_in_name and self._repo_keys and self._repo_keys != self._repo_in_name: 107 error( 108 f"Received inconsistent repos: '{self._repo_in_name}' and '{self._repo_keys}'." 109 ) 110 repo_connector = parse_repo_keys(repo_keys) 111 self._repo_connector = repo_connector 112 return self._repo_connector 113 114 115 @property 116 def version(self): 117 """ 118 Return the plugin's module version is defined (`__version__`) if it's defined. 119 """ 120 if self._version is None: 121 try: 122 self._version = self.module.__version__ 123 except Exception as e: 124 self._version = None 125 return self._version 126 127 128 @property 129 def module(self): 130 """ 131 Return the Python module of the underlying plugin, 132 or `None` if the plugin is not installed or its import failed 133 (check `Plugin.import_error` for the swallowed exception). 134 """ 135 if '_module' not in self.__dict__ or self.__dict__.get('_module', None) is None: 136 from meerschaum.plugins import _get_loaded_module, _get_plugins_package_name 137 loaded_module = _get_loaded_module(f'{_get_plugins_package_name()}.{self.name}') 138 if loaded_module is not None: 139 self._module = loaded_module 140 return self._module 141 142 if self.__file__ is None: 143 return None 144 145 from meerschaum.plugins import import_plugins 146 self._module = import_plugins(str(self), warn=False) 147 148 return self._module 149 150 @property 151 def import_error(self) -> Union[Exception, None]: 152 """ 153 Return the exception raised by this plugin's most recent failed import 154 (i.e. when `Plugin.module` is `None` even though the plugin is installed), 155 otherwise `None`. 156 """ 157 from meerschaum.plugins import _plugins_import_errors 158 return _plugins_import_errors.get(self.name, None) 159 160 161 @property 162 def __file__(self) -> Union[str, None]: 163 """ 164 Return the file path (str) of the plugin if it exists, otherwise `None`. 165 """ 166 if self.__dict__.get('_module', None) is not None: 167 return self.module.__file__ 168 169 import meerschaum.config.paths as paths 170 171 potential_dir = paths.PLUGINS_RESOURCES_PATH / self.name 172 if ( 173 potential_dir.exists() 174 and potential_dir.is_dir() 175 and (potential_dir / '__init__.py').exists() 176 ): 177 return str((potential_dir / '__init__.py').as_posix()) 178 179 potential_file = paths.PLUGINS_RESOURCES_PATH / (self.name + '.py') 180 if potential_file.exists() and not potential_file.is_dir(): 181 return str(potential_file.as_posix()) 182 183 return None 184 185 186 @property 187 def requirements_file_path(self) -> Union[pathlib.Path, None]: 188 """ 189 If a file named `requirements.txt` exists, return its path. 190 """ 191 if self.__file__ is None: 192 return None 193 path = pathlib.Path(self.__file__).parent / 'requirements.txt' 194 if not path.exists(): 195 return None 196 return path 197 198 199 def is_installed(self, **kw) -> bool: 200 """ 201 Check whether a plugin is correctly installed. 202 203 Returns 204 ------- 205 A `bool` indicating whether a plugin exists and is successfully imported. 206 """ 207 return self.__file__ is not None 208 209 210 def make_tar(self, debug: bool = False) -> pathlib.Path: 211 """ 212 Compress the plugin's source files into a `.tar.gz` archive and return the archive's path. 213 214 Parameters 215 ---------- 216 debug: bool, default False 217 Verbosity toggle. 218 219 Returns 220 ------- 221 A `pathlib.Path` to the archive file's path. 222 223 """ 224 import tarfile, pathlib, subprocess, fnmatch 225 from meerschaum.utils.debug import dprint 226 from meerschaum.utils.packages import attempt_import 227 pathspec = attempt_import('pathspec', debug=debug) 228 229 if not self.__file__: 230 from meerschaum.utils.warnings import error 231 error(f"Could not find file for plugin '{self}'.") 232 if '__init__.py' in self.__file__ or os.path.isdir(self.__file__): 233 path = self.__file__.replace('__init__.py', '') 234 is_dir = True 235 else: 236 path = self.__file__ 237 is_dir = False 238 239 old_cwd = os.getcwd() 240 real_parent_path = pathlib.Path(os.path.realpath(path)).parent 241 os.chdir(real_parent_path) 242 243 default_patterns_to_ignore = [ 244 '.pyc', 245 '__pycache__/', 246 'eggs/', 247 '__pypackages__/', 248 '.git', 249 ] 250 251 def parse_gitignore() -> 'Set[str]': 252 gitignore_path = pathlib.Path(path) / '.gitignore' 253 if not gitignore_path.exists(): 254 return set(default_patterns_to_ignore) 255 with open(gitignore_path, 'r', encoding='utf-8') as f: 256 gitignore_text = f.read() 257 return set(pathspec.PathSpec.from_lines( 258 pathspec.patterns.GitWildMatchPattern, 259 default_patterns_to_ignore + gitignore_text.splitlines() 260 ).match_tree(path)) 261 262 patterns_to_ignore = parse_gitignore() if is_dir else set() 263 264 if debug: 265 dprint(f"Patterns to ignore:\n{patterns_to_ignore}") 266 267 with tarfile.open(self.archive_path, 'w:gz') as tarf: 268 if not is_dir: 269 tarf.add(f"{self.name}.py") 270 else: 271 for root, dirs, files in os.walk(self.name): 272 for f in files: 273 good_file = True 274 fp = os.path.join(root, f) 275 for pattern in patterns_to_ignore: 276 if pattern in str(fp) or f.startswith('.'): 277 good_file = False 278 break 279 if good_file: 280 if debug: 281 dprint(f"Adding '{fp}'...") 282 tarf.add(fp) 283 284 ### clean up and change back to old directory 285 os.chdir(old_cwd) 286 287 ### change to 775 to avoid permissions issues with the API in a Docker container 288 self.archive_path.chmod(0o775) 289 290 if debug: 291 dprint(f"Created archive '{self.archive_path}'.") 292 return self.archive_path 293 294 295 def install( 296 self, 297 skip_deps: bool = False, 298 force: bool = False, 299 debug: bool = False, 300 ) -> SuccessTuple: 301 """ 302 Extract a plugin's tar archive to the plugins directory. 303 304 This function checks if the plugin is already installed and if the version is equal or 305 greater than the existing installation. 306 307 Parameters 308 ---------- 309 skip_deps: bool, default False 310 If `True`, do not install dependencies. 311 312 force: bool, default False 313 If `True`, continue with installation, even if required packages fail to install. 314 315 debug: bool, default False 316 Verbosity toggle. 317 318 Returns 319 ------- 320 A `SuccessTuple` of success (bool) and a message (str). 321 322 """ 323 installation_key = self.full_name 324 with _ongoing_installations_lock: 325 if installation_key in _ongoing_installations: 326 return True, f"Already installing plugin '{self}'." 327 _ongoing_installations.add(installation_key) 328 329 try: 330 from meerschaum.utils.packages import _pip_install_lock 331 with _pip_install_lock(self.name): 332 return self._install_with_rollback( 333 skip_deps=skip_deps, 334 force=force, 335 debug=debug, 336 ) 337 finally: 338 with _ongoing_installations_lock: 339 _ongoing_installations.discard(installation_key) 340 341 342 def _install_with_rollback( 343 self, 344 skip_deps: bool = False, 345 force: bool = False, 346 debug: bool = False, 347 ) -> SuccessTuple: 348 """Restore the previous plugin source and environment after a failed install.""" 349 import tempfile 350 import meerschaum.config.paths as paths 351 from meerschaum.plugins import sync_plugins_symlinks 352 from meerschaum.utils.packages import reload_meerschaum 353 354 if not self.archive_path.exists(): 355 return False, f"Missing archive file for plugin '{self}'." 356 357 paths.PLUGINS_TEMP_RESOURCES_PATH.mkdir(parents=True, exist_ok=True) 358 source_paths = [ 359 plugins_dir_path / filename 360 for plugins_dir_path in paths.PLUGINS_DIR_PATHS 361 for filename in (self.name, self.name + '.py') 362 ] 363 tracked_paths = source_paths + [paths.VIRTENV_RESOURCES_PATH / self.name] 364 365 with tempfile.TemporaryDirectory( 366 prefix=f'.{self.name}-rollback-', 367 dir=paths.PLUGINS_TEMP_RESOURCES_PATH, 368 ) as rollback_dir_str: 369 rollback_dir = pathlib.Path(rollback_dir_str) 370 backups = {} 371 rollback_prepared = False 372 373 def prepare_rollback() -> None: 374 nonlocal rollback_prepared 375 for index, path in enumerate(tracked_paths): 376 if not path.exists() and not path.is_symlink(): 377 continue 378 backup_path = rollback_dir / str(index) 379 _copy_path(path, backup_path) 380 backups[path] = backup_path 381 rollback_prepared = True 382 383 def rollback() -> None: 384 if not rollback_prepared: 385 return 386 for path in tracked_paths: 387 _remove_path(path) 388 for path, backup_path in backups.items(): 389 path.parent.mkdir(parents=True, exist_ok=True) 390 shutil.move(str(backup_path), str(path)) 391 self.__dict__.pop('_module', None) 392 self.__dict__.pop('_required', None) 393 sync_plugins_symlinks(debug=debug) 394 reload_meerschaum(debug=debug) 395 396 try: 397 result = self._install( 398 skip_deps=skip_deps, 399 force=force, 400 debug=debug, 401 _prepare_rollback=prepare_rollback, 402 ) 403 except BaseException: 404 rollback() 405 raise 406 if not result[0]: 407 try: 408 rollback() 409 except Exception as e: 410 return False, result[1] + f"\nFailed to roll back plugin '{self}': {e}" 411 return result 412 413 414 def _install( 415 self, 416 skip_deps: bool = False, 417 force: bool = False, 418 debug: bool = False, 419 _prepare_rollback: Optional[Any] = None, 420 ) -> SuccessTuple: 421 422 import meerschaum.config.paths as paths 423 from meerschaum.utils.warnings import warn, error 424 if debug: 425 from meerschaum.utils.debug import dprint 426 import tarfile 427 import re 428 import ast 429 from meerschaum.plugins import sync_plugins_symlinks 430 from meerschaum.utils.packages import attempt_import, reload_meerschaum 431 from meerschaum.utils.venv import init_venv 432 from meerschaum.utils.misc import safely_extract_tar 433 old_cwd = os.getcwd() 434 old_version = '' 435 new_version = '' 436 temp_dir = paths.PLUGINS_TEMP_RESOURCES_PATH / self.name 437 temp_dir.mkdir(exist_ok=True) 438 439 if not self.archive_path.exists(): 440 return False, f"Missing archive file for plugin '{self}'." 441 if self.version is not None: 442 old_version = self.version 443 if debug: 444 dprint(f"Found existing version '{old_version}' for plugin '{self}'.") 445 446 if debug: 447 dprint(f"Extracting '{self.archive_path}' to '{temp_dir}'...") 448 449 try: 450 with tarfile.open(self.archive_path, 'r:gz') as tarf: 451 safely_extract_tar(tarf, temp_dir) 452 except Exception as e: 453 warn(e) 454 return False, f"Failed to extract plugin '{self.name}'." 455 456 ### search for version information 457 files = os.listdir(temp_dir) 458 459 if str(files[0]) == self.name: 460 is_dir = True 461 elif str(files[0]) == self.name + '.py': 462 is_dir = False 463 else: 464 error(f"Unknown format encountered for plugin '{self}'.") 465 466 fpath = temp_dir / files[0] 467 if is_dir: 468 fpath = fpath / '__init__.py' 469 470 with open(fpath, 'r', encoding='utf-8') as f: 471 init_lines = f.readlines() 472 new_version = None 473 for line in init_lines: 474 if '__version__' not in line: 475 continue 476 version_match = re.search(r'__version__(\s?)=', line.lstrip().rstrip()) 477 if not version_match: 478 continue 479 new_version = ast.literal_eval(line.split('=')[1].lstrip().rstrip()) 480 break 481 if not new_version: 482 warn( 483 f"No `__version__` defined for plugin '{self}'. " 484 + "Assuming new version...", 485 stack = False, 486 ) 487 488 packaging_version = attempt_import('packaging.version') 489 try: 490 is_new_version = (not new_version and not old_version) or ( 491 packaging_version.parse(old_version) < packaging_version.parse(new_version) 492 ) 493 is_same_version = new_version and old_version and ( 494 packaging_version.parse(old_version) == packaging_version.parse(new_version) 495 ) 496 except Exception: 497 is_new_version, is_same_version = True, False 498 499 ### Determine where to permanently store the new plugin. 500 plugin_installation_dir_path = paths.PLUGINS_DIR_PATHS[0] 501 for path in paths.PLUGINS_DIR_PATHS: 502 if not path.exists(): 503 warn(f"Plugins path does not exist: {path}", stack=False) 504 continue 505 506 files_in_plugins_dir = os.listdir(path) 507 if ( 508 self.name in files_in_plugins_dir 509 or 510 (self.name + '.py') in files_in_plugins_dir 511 ): 512 plugin_installation_dir_path = path 513 break 514 515 success_msg = ( 516 f"Successfully installed plugin '{self}'" 517 + ("\n (skipped dependencies)" if skip_deps else "") 518 + "." 519 ) 520 success, abort = None, None 521 522 if is_same_version and not force: 523 success, msg = True, ( 524 f"Plugin '{self}' is up-to-date (version {old_version}).\n" + 525 " Install again with `-f` or `--force` to reinstall." 526 ) 527 abort = True 528 elif is_new_version or force: 529 if _prepare_rollback is not None: 530 _prepare_rollback() 531 source_path = temp_dir / files[0] 532 destination_path = plugin_installation_dir_path / files[0] 533 try: 534 plugin_installation_dir_path.mkdir(parents=True, exist_ok=True) 535 for existing_path in ( 536 plugin_installation_dir_path / self.name, 537 plugin_installation_dir_path / (self.name + '.py'), 538 ): 539 _remove_path(existing_path) 540 if debug: 541 dprint(f"Moving '{source_path}' to '{destination_path}'...") 542 shutil.move(str(source_path), str(destination_path)) 543 except Exception as e: 544 success, msg = False, f"Failed to install plugin '{self}': {e}" 545 else: 546 success, msg = True, success_msg 547 else: 548 success, msg = False, ( 549 f"Your installed version of plugin '{self}' ({old_version}) is higher than " 550 + f"attempted version {new_version}." 551 ) 552 553 shutil.rmtree(temp_dir) 554 os.chdir(old_cwd) 555 556 ### Reload the plugin's module. 557 sync_plugins_symlinks(debug=debug) 558 if '_module' in self.__dict__: 559 del self.__dict__['_module'] 560 init_venv(venv=self.name, force=True, debug=debug) 561 reload_meerschaum(debug=debug) 562 563 ### Record the origin only after a successful transaction. Locally-installed 564 ### plugins have no explicit repo and are intentionally skipped. 565 _origin_repo_keys = ( 566 str(self._repo_connector) 567 if self._repo_connector is not None 568 else self._repo_keys or self._repo_in_name 569 ) 570 571 def write_origin() -> None: 572 if not _origin_repo_keys: 573 return 574 try: 575 from meerschaum.plugins._origins import write_plugin_origin 576 write_plugin_origin( 577 self.name, 578 _origin_repo_keys, 579 plugin_installation_dir_path, 580 debug=debug, 581 ) 582 except Exception: 583 pass 584 585 if abort: 586 write_origin() 587 588 ### if we've already failed, return here 589 if not success or abort: 590 return success, msg 591 592 ### attempt to install dependencies 593 dependencies_installed = skip_deps or self.install_dependencies(force=force, debug=debug) 594 if not dependencies_installed: 595 return False, f"Failed to install dependencies for plugin '{self}'." 596 597 ### handling success tuple, bool, or other (typically None) 598 setup_tuple = self.setup(debug=debug) 599 if isinstance(setup_tuple, tuple): 600 if not setup_tuple[0]: 601 success, msg = setup_tuple 602 elif isinstance(setup_tuple, bool): 603 if not setup_tuple: 604 success, msg = False, ( 605 f"Failed to run post-install setup for plugin '{self}'." + '\n' + 606 f"Check `setup()` in '{self.__file__}' for more information " + 607 "(no error message provided)." 608 ) 609 else: 610 success, msg = True, success_msg 611 elif setup_tuple is None: 612 success = True 613 msg = ( 614 f"Post-install for plugin '{self}' returned None. " + 615 "Assuming plugin successfully installed." 616 ) 617 warn(msg) 618 else: 619 success = False 620 msg = ( 621 f"Post-install for plugin '{self}' returned unexpected value " + 622 f"of type '{type(setup_tuple)}': {setup_tuple}" 623 ) 624 625 _ = self.module 626 if success: 627 write_origin() 628 return success, msg 629 630 631 def remove_archive( 632 self, 633 debug: bool = False 634 ) -> SuccessTuple: 635 """Remove a plugin's archive file.""" 636 if not self.archive_path.exists(): 637 return True, f"Archive file for plugin '{self}' does not exist." 638 try: 639 self.archive_path.unlink() 640 except Exception as e: 641 return False, f"Failed to remove archive for plugin '{self}':\n{e}" 642 return True, "Success" 643 644 645 def remove_venv( 646 self, 647 debug: bool = False 648 ) -> SuccessTuple: 649 """Remove a plugin's virtual environment.""" 650 if not self.venv_path.exists(): 651 return True, f"Virtual environment for plugin '{self}' does not exist." 652 try: 653 shutil.rmtree(self.venv_path) 654 except Exception as e: 655 return False, f"Failed to remove virtual environment for plugin '{self}':\n{e}" 656 return True, "Success" 657 658 659 def uninstall(self, debug: bool = False) -> SuccessTuple: 660 """ 661 Remove a plugin, its virtual environment, and archive file. 662 """ 663 from meerschaum.utils.packages import reload_meerschaum 664 from meerschaum.plugins import sync_plugins_symlinks 665 from meerschaum.utils.warnings import warn, info 666 warnings_thrown_count: int = 0 667 max_warnings: int = 3 668 669 if not self.is_installed(): 670 info( 671 f"Plugin '{self.name}' doesn't seem to be installed.\n " 672 + "Checking for artifacts...", 673 stack = False, 674 ) 675 else: 676 real_path = pathlib.Path(os.path.realpath(self.__file__)) 677 try: 678 if real_path.name == '__init__.py': 679 shutil.rmtree(real_path.parent) 680 else: 681 real_path.unlink() 682 except Exception as e: 683 warn(f"Could not remove source files for plugin '{self.name}':\n{e}", stack=False) 684 warnings_thrown_count += 1 685 else: 686 info(f"Removed source files for plugin '{self.name}'.") 687 688 if self.venv_path.exists(): 689 success, msg = self.remove_venv(debug=debug) 690 if not success: 691 warn(msg, stack=False) 692 warnings_thrown_count += 1 693 else: 694 info(f"Removed virtual environment from plugin '{self.name}'.") 695 696 success = warnings_thrown_count < max_warnings 697 try: 698 from meerschaum.plugins._origins import remove_plugin_origin 699 remove_plugin_origin(self.name, debug=debug) 700 except Exception: 701 pass 702 sync_plugins_symlinks(debug=debug) 703 self.deactivate_venv(force=True, debug=debug) 704 reload_meerschaum(debug=debug) 705 return success, ( 706 f"Successfully uninstalled plugin '{self}'." if success 707 else f"Failed to uninstall plugin '{self}'." 708 ) 709 710 711 def setup(self, *args: str, debug: bool = False, **kw: Any) -> Union[SuccessTuple, bool]: 712 """ 713 If exists, run the plugin's `setup()` function. 714 715 Parameters 716 ---------- 717 *args: str 718 The positional arguments passed to the `setup()` function. 719 720 debug: bool, default False 721 Verbosity toggle. 722 723 **kw: Any 724 The keyword arguments passed to the `setup()` function. 725 726 Returns 727 ------- 728 A `SuccessTuple` or `bool` indicating success. 729 730 """ 731 from meerschaum.utils.debug import dprint 732 import inspect 733 _setup = None 734 for name, fp in inspect.getmembers(self.module): 735 if name == 'setup' and inspect.isfunction(fp): 736 _setup = fp 737 break 738 739 ### assume success if no setup() is found (not necessary) 740 if _setup is None: 741 return True 742 743 sig = inspect.signature(_setup) 744 has_debug, has_kw = ('debug' in sig.parameters), False 745 for k, v in sig.parameters.items(): 746 if '**' in str(v): 747 has_kw = True 748 break 749 750 _kw = {} 751 if has_kw: 752 _kw.update(kw) 753 if has_debug: 754 _kw['debug'] = debug 755 756 if debug: 757 dprint(f"Running setup for plugin '{self}'...") 758 from meerschaum.utils.venv import Venv 759 try: 760 with Venv(self, debug=debug): 761 return_tuple = _setup(*args, **_kw) 762 except Exception as e: 763 return False, str(e) 764 765 if isinstance(return_tuple, tuple): 766 return return_tuple 767 if isinstance(return_tuple, bool): 768 return return_tuple, f"Setup for Plugin '{self.name}' did not return a message." 769 if return_tuple is None: 770 return False, f"Setup for Plugin '{self.name}' returned None." 771 return False, f"Unknown return value from setup for Plugin '{self.name}': {return_tuple}" 772 773 774 def get_dependencies( 775 self, 776 debug: bool = False, 777 ) -> List[str]: 778 """ 779 If the Plugin has specified dependencies in a list called `required`, return the list. 780 781 **NOTE:** Dependecies which start with `'plugin:'` are Meerschaum plugins, not pip packages. 782 Meerschaum plugins may also specify connector keys for a repo after `'@'`. 783 784 Parameters 785 ---------- 786 debug: bool, default False 787 Verbosity toggle. 788 789 Returns 790 ------- 791 A list of required packages and plugins (str). 792 793 """ 794 if '_required' in self.__dict__: 795 return self._required 796 797 ### If the plugin has not yet been imported, 798 ### infer the dependencies from the source text. 799 ### This is not super robust, and it doesn't feel right 800 ### having multiple versions of the logic. 801 ### This is necessary when determining the activation order 802 ### without having import the module. 803 ### For consistency's sake, the module-less method does not cache the requirements. 804 if self.__dict__.get('_module', None) is None: 805 file_path = self.__file__ 806 if file_path is None: 807 return [] 808 with open(file_path, 'r', encoding='utf-8') as f: 809 text = f.read() 810 811 if 'required' not in text: 812 return [] 813 814 ### This has some limitations: 815 ### It relies on `required` being manually declared. 816 ### We lose the ability to dynamically alter the `required` list, 817 ### which is why we've kept the module-reliant method below. 818 import ast, re 819 ### NOTE: This technically would break 820 ### if `required` was the very first line of the file. 821 req_start_match = re.search(r'\nrequired(:\s*)?.*=', text) 822 if not req_start_match: 823 return [] 824 req_start = req_start_match.start() 825 equals_sign = req_start + text[req_start:].find('=') 826 827 ### Dependencies may have brackets within the strings, so push back the index. 828 first_opening_brace = equals_sign + 1 + text[equals_sign:].find('[') 829 if first_opening_brace == -1: 830 return [] 831 832 next_closing_brace = equals_sign + 1 + text[equals_sign:].find(']') 833 if next_closing_brace == -1: 834 return [] 835 836 start_ix = first_opening_brace + 1 837 end_ix = next_closing_brace 838 839 num_braces = 0 840 while True: 841 if '[' not in text[start_ix:end_ix]: 842 break 843 num_braces += 1 844 start_ix = end_ix 845 end_ix += text[end_ix + 1:].find(']') + 1 846 847 req_end = end_ix + 1 848 req_text = ( 849 text[(first_opening_brace-1):req_end] 850 .lstrip() 851 .replace('=', '', 1) 852 .lstrip() 853 .rstrip() 854 ) 855 try: 856 required = ast.literal_eval(req_text) 857 except Exception as e: 858 warn( 859 f"Unable to determine requirements for plugin '{self.name}' " 860 + "without importing the module.\n" 861 + " This may be due to dynamically setting the global `required` list.\n" 862 + f" {e}" 863 ) 864 return [] 865 return required 866 867 import inspect 868 self.activate_venv(dependencies=False, debug=debug) 869 required = [] 870 for name, val in inspect.getmembers(self.module): 871 if name == 'required': 872 required = val 873 break 874 self._required = required 875 self.deactivate_venv(dependencies=False, debug=debug) 876 return required 877 878 879 def get_required_plugins(self, debug: bool=False) -> List[mrsm.plugins.Plugin]: 880 """ 881 Return a list of required Plugin objects. 882 """ 883 from meerschaum.utils.warnings import warn 884 from meerschaum.config import get_config 885 from meerschaum._internal.static import STATIC_CONFIG 886 from meerschaum.connectors.parse import is_valid_connector_keys 887 plugins = [] 888 _deps = self.get_dependencies(debug=debug) 889 sep = STATIC_CONFIG['plugins']['repo_separator'] 890 plugin_names = [ 891 _d[len('plugin:'):] for _d in _deps 892 if _d.startswith('plugin:') and len(_d) > len('plugin:') 893 ] 894 default_repo_keys = get_config('meerschaum', 'repository') 895 skipped_repo_keys = set() 896 897 for _plugin_name in plugin_names: 898 if sep in _plugin_name: 899 try: 900 _plugin_name, _repo_keys = _plugin_name.split(sep) 901 except Exception: 902 _repo_keys = default_repo_keys 903 warn( 904 f"Invalid repo keys for required plugin '{_plugin_name}'.\n " 905 + f"Will try to use '{_repo_keys}' instead.", 906 stack = False, 907 ) 908 else: 909 _repo_keys = default_repo_keys 910 911 if _repo_keys in skipped_repo_keys: 912 continue 913 914 if not is_valid_connector_keys(_repo_keys): 915 warn( 916 f"Invalid connector '{_repo_keys}'.\n" 917 f" Skipping required plugins from repository '{_repo_keys}'", 918 stack=False, 919 ) 920 continue 921 922 plugins.append(Plugin(_plugin_name, repo=_repo_keys)) 923 924 return plugins 925 926 927 def get_required_packages(self, debug: bool=False) -> List[str]: 928 """ 929 Return the required package names (excluding plugins). 930 """ 931 _deps = self.get_dependencies(debug=debug) 932 return [_d for _d in _deps if not _d.startswith('plugin:')] 933 934 935 def activate_venv( 936 self, 937 dependencies: bool = True, 938 init_if_not_exists: bool = True, 939 debug: bool = False, 940 **kw 941 ) -> bool: 942 """ 943 Activate the virtual environments for the plugin and its dependencies. 944 945 Parameters 946 ---------- 947 dependencies: bool, default True 948 If `True`, activate the virtual environments for required plugins. 949 950 Returns 951 ------- 952 A bool indicating success. 953 """ 954 import meerschaum.config.paths as paths 955 from meerschaum.utils.venv import venv_target_path 956 from meerschaum.utils.packages import activate_venv 957 from meerschaum.utils.misc import make_symlink, is_symlink 958 959 if dependencies: 960 for plugin in self.get_required_plugins(debug=debug): 961 plugin.activate_venv(debug=debug, init_if_not_exists=init_if_not_exists, **kw) 962 963 vtp = venv_target_path(self.name, debug=debug, allow_nonexistent=True) 964 venv_meerschaum_path = vtp / 'meerschaum' 965 966 try: 967 success, msg = True, "Success" 968 if is_symlink(venv_meerschaum_path): 969 if pathlib.Path(os.path.realpath(venv_meerschaum_path)) != paths.PACKAGE_ROOT_PATH: 970 venv_meerschaum_path.unlink() 971 success, msg = make_symlink(venv_meerschaum_path, paths.PACKAGE_ROOT_PATH) 972 except Exception as e: 973 success, msg = False, str(e) 974 if not success: 975 warn( 976 f"Unable to create symlink {venv_meerschaum_path} to {paths.PACKAGE_ROOT_PATH}:\n" 977 f"{msg}" 978 ) 979 980 return activate_venv(self.name, init_if_not_exists=init_if_not_exists, debug=debug, **kw) 981 982 983 def deactivate_venv(self, dependencies: bool=True, debug: bool = False, **kw) -> bool: 984 """ 985 Deactivate the virtual environments for the plugin and its dependencies. 986 987 Parameters 988 ---------- 989 dependencies: bool, default True 990 If `True`, deactivate the virtual environments for required plugins. 991 992 Returns 993 ------- 994 A bool indicating success. 995 """ 996 from meerschaum.utils.packages import deactivate_venv 997 success = deactivate_venv(self.name, debug=debug, **kw) 998 if dependencies: 999 for plugin in self.get_required_plugins(debug=debug): 1000 plugin.deactivate_venv(debug=debug, **kw) 1001 return success 1002 1003 1004 def install_dependencies( 1005 self, 1006 force: bool = False, 1007 debug: bool = False, 1008 ) -> bool: 1009 """ 1010 If specified, install dependencies. 1011 1012 **NOTE:** Dependencies that start with `'plugin:'` will be installed as 1013 Meerschaum plugins from the same repository as this Plugin. 1014 To install from a different repository, add the repo keys after `'@'` 1015 (e.g. `'plugin:foo@api:bar'`). 1016 1017 Parameters 1018 ---------- 1019 force: bool, default False 1020 If `True`, continue with the installation, even if some 1021 required packages fail to install. 1022 1023 debug: bool, default False 1024 Verbosity toggle. 1025 1026 Returns 1027 ------- 1028 A bool indicating success. 1029 """ 1030 from meerschaum.utils.packages import pip_install, venv_contains_package 1031 from meerschaum.utils.warnings import warn, info 1032 _deps = self.get_dependencies(debug=debug) 1033 if not _deps and self.requirements_file_path is None: 1034 return True 1035 1036 plugins = self.get_required_plugins(debug=debug) 1037 for _plugin in plugins: 1038 if _plugin.name == self.name: 1039 warn(f"Plugin '{self.name}' cannot depend on itself! Skipping...", stack=False) 1040 continue 1041 _success, _msg = _plugin.repo_connector.install_plugin( 1042 _plugin.name, debug=debug, force=force 1043 ) 1044 if not _success: 1045 warn( 1046 f"Failed to install required plugin '{_plugin}' from '{_plugin.repo_connector}'" 1047 + f" for plugin '{self.name}':\n" + _msg, 1048 stack = False, 1049 ) 1050 if not force: 1051 warn( 1052 "Try installing with the `--force` flag to continue anyway.", 1053 stack = False, 1054 ) 1055 return False 1056 info( 1057 "Continuing with installation despite the failure " 1058 + "(careful, things might be broken!)...", 1059 icon = False 1060 ) 1061 1062 1063 ### First step: parse `requirements.txt` if it exists. 1064 if self.requirements_file_path is not None: 1065 if not pip_install( 1066 requirements_file_path=self.requirements_file_path, 1067 venv=self.name, debug=debug 1068 ): 1069 warn( 1070 f"Failed to resolve 'requirements.txt' for plugin '{self.name}'.", 1071 stack = False, 1072 ) 1073 if not force: 1074 warn( 1075 "Try installing with `--force` to continue anyway.", 1076 stack = False, 1077 ) 1078 return False 1079 info( 1080 "Continuing with installation despite the failure " 1081 + "(careful, things might be broken!)...", 1082 icon = False 1083 ) 1084 1085 1086 ### Don't reinstall packages that are already included in required plugins. 1087 packages = [] 1088 _packages = self.get_required_packages(debug=debug) 1089 accounted_for_packages = set() 1090 for package_name in _packages: 1091 for plugin in plugins: 1092 if venv_contains_package(package_name, plugin.name): 1093 accounted_for_packages.add(package_name) 1094 break 1095 packages = [pkg for pkg in _packages if pkg not in accounted_for_packages] 1096 1097 ### Attempt pip packages installation. 1098 if packages: 1099 for package in packages: 1100 if not pip_install(package, venv=self.name, debug=debug): 1101 warn( 1102 f"Failed to install required package '{package}'" 1103 + f" for plugin '{self.name}'.", 1104 stack = False, 1105 ) 1106 if not force: 1107 warn( 1108 "Try installing with `--force` to continue anyway.", 1109 stack = False, 1110 ) 1111 return False 1112 info( 1113 "Continuing with installation despite the failure " 1114 + "(careful, things might be broken!)...", 1115 icon = False 1116 ) 1117 return True 1118 1119 1120 @property 1121 def full_name(self) -> str: 1122 """ 1123 Include the repo keys with the plugin's name. 1124 """ 1125 from meerschaum._internal.static import STATIC_CONFIG 1126 sep = STATIC_CONFIG['plugins']['repo_separator'] 1127 return self.name + sep + str(self.repo_connector) 1128 1129 1130 def __str__(self): 1131 return self.name 1132 1133 1134 def __repr__(self): 1135 return f"Plugin('{self.name}', repo='{self.repo_connector}')" 1136 1137 1138 def __del__(self): 1139 pass
Handle packaging of Meerschaum plugins.
53 def __init__( 54 self, 55 name: str, 56 version: Optional[str] = None, 57 user_id: Optional[int] = None, 58 required: Optional[List[str]] = None, 59 attributes: Optional[Dict[str, Any]] = None, 60 archive_path: Optional[pathlib.Path] = None, 61 venv_path: Optional[pathlib.Path] = None, 62 repo_connector: Optional['mrsm.connectors.api.APIConnector'] = None, 63 repo: Union['mrsm.connectors.api.APIConnector', str, None] = None, 64 ): 65 import meerschaum.config.paths as paths 66 from meerschaum._internal.static import STATIC_CONFIG 67 sep = STATIC_CONFIG['plugins']['repo_separator'] 68 _repo = None 69 if sep in name: 70 try: 71 name, _repo = name.split(sep) 72 except Exception as e: 73 error(f"Invalid plugin name: '{name}'") 74 self._repo_in_name = _repo 75 76 if attributes is None: 77 attributes = {} 78 self.name = name 79 self.attributes = attributes 80 self.user_id = user_id 81 self._version = version 82 if required: 83 self._required = required 84 self.archive_path = ( 85 archive_path if archive_path is not None 86 else paths.PLUGINS_ARCHIVES_RESOURCES_PATH / f"{self.name}.tar.gz" 87 ) 88 self.venv_path = ( 89 venv_path if venv_path is not None 90 else paths.VIRTENV_RESOURCES_PATH / self.name 91 ) 92 self._repo_connector = repo_connector 93 self._repo_keys = repo
96 @property 97 def repo_connector(self): 98 """ 99 Return the repository connector for this plugin. 100 NOTE: This imports the `connectors` module, which imports certain plugin modules. 101 """ 102 if self._repo_connector is None: 103 from meerschaum.connectors.parse import parse_repo_keys 104 105 repo_keys = self._repo_keys or self._repo_in_name 106 if self._repo_in_name and self._repo_keys and self._repo_keys != self._repo_in_name: 107 error( 108 f"Received inconsistent repos: '{self._repo_in_name}' and '{self._repo_keys}'." 109 ) 110 repo_connector = parse_repo_keys(repo_keys) 111 self._repo_connector = repo_connector 112 return self._repo_connector
Return the repository connector for this plugin.
NOTE: This imports the connectors module, which imports certain plugin modules.
115 @property 116 def version(self): 117 """ 118 Return the plugin's module version is defined (`__version__`) if it's defined. 119 """ 120 if self._version is None: 121 try: 122 self._version = self.module.__version__ 123 except Exception as e: 124 self._version = None 125 return self._version
Return the plugin's module version is defined (__version__) if it's defined.
128 @property 129 def module(self): 130 """ 131 Return the Python module of the underlying plugin, 132 or `None` if the plugin is not installed or its import failed 133 (check `Plugin.import_error` for the swallowed exception). 134 """ 135 if '_module' not in self.__dict__ or self.__dict__.get('_module', None) is None: 136 from meerschaum.plugins import _get_loaded_module, _get_plugins_package_name 137 loaded_module = _get_loaded_module(f'{_get_plugins_package_name()}.{self.name}') 138 if loaded_module is not None: 139 self._module = loaded_module 140 return self._module 141 142 if self.__file__ is None: 143 return None 144 145 from meerschaum.plugins import import_plugins 146 self._module = import_plugins(str(self), warn=False) 147 148 return self._module
Return the Python module of the underlying plugin,
or None if the plugin is not installed or its import failed
(check Plugin.import_error for the swallowed exception).
150 @property 151 def import_error(self) -> Union[Exception, None]: 152 """ 153 Return the exception raised by this plugin's most recent failed import 154 (i.e. when `Plugin.module` is `None` even though the plugin is installed), 155 otherwise `None`. 156 """ 157 from meerschaum.plugins import _plugins_import_errors 158 return _plugins_import_errors.get(self.name, None)
Return the exception raised by this plugin's most recent failed import
(i.e. when Plugin.module is None even though the plugin is installed),
otherwise None.
186 @property 187 def requirements_file_path(self) -> Union[pathlib.Path, None]: 188 """ 189 If a file named `requirements.txt` exists, return its path. 190 """ 191 if self.__file__ is None: 192 return None 193 path = pathlib.Path(self.__file__).parent / 'requirements.txt' 194 if not path.exists(): 195 return None 196 return path
If a file named requirements.txt exists, return its path.
199 def is_installed(self, **kw) -> bool: 200 """ 201 Check whether a plugin is correctly installed. 202 203 Returns 204 ------- 205 A `bool` indicating whether a plugin exists and is successfully imported. 206 """ 207 return self.__file__ is not None
Check whether a plugin is correctly installed.
Returns
- A
boolindicating whether a plugin exists and is successfully imported.
210 def make_tar(self, debug: bool = False) -> pathlib.Path: 211 """ 212 Compress the plugin's source files into a `.tar.gz` archive and return the archive's path. 213 214 Parameters 215 ---------- 216 debug: bool, default False 217 Verbosity toggle. 218 219 Returns 220 ------- 221 A `pathlib.Path` to the archive file's path. 222 223 """ 224 import tarfile, pathlib, subprocess, fnmatch 225 from meerschaum.utils.debug import dprint 226 from meerschaum.utils.packages import attempt_import 227 pathspec = attempt_import('pathspec', debug=debug) 228 229 if not self.__file__: 230 from meerschaum.utils.warnings import error 231 error(f"Could not find file for plugin '{self}'.") 232 if '__init__.py' in self.__file__ or os.path.isdir(self.__file__): 233 path = self.__file__.replace('__init__.py', '') 234 is_dir = True 235 else: 236 path = self.__file__ 237 is_dir = False 238 239 old_cwd = os.getcwd() 240 real_parent_path = pathlib.Path(os.path.realpath(path)).parent 241 os.chdir(real_parent_path) 242 243 default_patterns_to_ignore = [ 244 '.pyc', 245 '__pycache__/', 246 'eggs/', 247 '__pypackages__/', 248 '.git', 249 ] 250 251 def parse_gitignore() -> 'Set[str]': 252 gitignore_path = pathlib.Path(path) / '.gitignore' 253 if not gitignore_path.exists(): 254 return set(default_patterns_to_ignore) 255 with open(gitignore_path, 'r', encoding='utf-8') as f: 256 gitignore_text = f.read() 257 return set(pathspec.PathSpec.from_lines( 258 pathspec.patterns.GitWildMatchPattern, 259 default_patterns_to_ignore + gitignore_text.splitlines() 260 ).match_tree(path)) 261 262 patterns_to_ignore = parse_gitignore() if is_dir else set() 263 264 if debug: 265 dprint(f"Patterns to ignore:\n{patterns_to_ignore}") 266 267 with tarfile.open(self.archive_path, 'w:gz') as tarf: 268 if not is_dir: 269 tarf.add(f"{self.name}.py") 270 else: 271 for root, dirs, files in os.walk(self.name): 272 for f in files: 273 good_file = True 274 fp = os.path.join(root, f) 275 for pattern in patterns_to_ignore: 276 if pattern in str(fp) or f.startswith('.'): 277 good_file = False 278 break 279 if good_file: 280 if debug: 281 dprint(f"Adding '{fp}'...") 282 tarf.add(fp) 283 284 ### clean up and change back to old directory 285 os.chdir(old_cwd) 286 287 ### change to 775 to avoid permissions issues with the API in a Docker container 288 self.archive_path.chmod(0o775) 289 290 if debug: 291 dprint(f"Created archive '{self.archive_path}'.") 292 return self.archive_path
Compress the plugin's source files into a .tar.gz archive and return the archive's path.
Parameters
- debug (bool, default False): Verbosity toggle.
Returns
- A
pathlib.Pathto the archive file's path.
295 def install( 296 self, 297 skip_deps: bool = False, 298 force: bool = False, 299 debug: bool = False, 300 ) -> SuccessTuple: 301 """ 302 Extract a plugin's tar archive to the plugins directory. 303 304 This function checks if the plugin is already installed and if the version is equal or 305 greater than the existing installation. 306 307 Parameters 308 ---------- 309 skip_deps: bool, default False 310 If `True`, do not install dependencies. 311 312 force: bool, default False 313 If `True`, continue with installation, even if required packages fail to install. 314 315 debug: bool, default False 316 Verbosity toggle. 317 318 Returns 319 ------- 320 A `SuccessTuple` of success (bool) and a message (str). 321 322 """ 323 installation_key = self.full_name 324 with _ongoing_installations_lock: 325 if installation_key in _ongoing_installations: 326 return True, f"Already installing plugin '{self}'." 327 _ongoing_installations.add(installation_key) 328 329 try: 330 from meerschaum.utils.packages import _pip_install_lock 331 with _pip_install_lock(self.name): 332 return self._install_with_rollback( 333 skip_deps=skip_deps, 334 force=force, 335 debug=debug, 336 ) 337 finally: 338 with _ongoing_installations_lock: 339 _ongoing_installations.discard(installation_key)
Extract a plugin's tar archive to the plugins directory.
This function checks if the plugin is already installed and if the version is equal or greater than the existing installation.
Parameters
- skip_deps (bool, default False):
If
True, do not install dependencies. - force (bool, default False):
If
True, continue with installation, even if required packages fail to install. - debug (bool, default False): Verbosity toggle.
Returns
- A
SuccessTupleof success (bool) and a message (str).
631 def remove_archive( 632 self, 633 debug: bool = False 634 ) -> SuccessTuple: 635 """Remove a plugin's archive file.""" 636 if not self.archive_path.exists(): 637 return True, f"Archive file for plugin '{self}' does not exist." 638 try: 639 self.archive_path.unlink() 640 except Exception as e: 641 return False, f"Failed to remove archive for plugin '{self}':\n{e}" 642 return True, "Success"
Remove a plugin's archive file.
645 def remove_venv( 646 self, 647 debug: bool = False 648 ) -> SuccessTuple: 649 """Remove a plugin's virtual environment.""" 650 if not self.venv_path.exists(): 651 return True, f"Virtual environment for plugin '{self}' does not exist." 652 try: 653 shutil.rmtree(self.venv_path) 654 except Exception as e: 655 return False, f"Failed to remove virtual environment for plugin '{self}':\n{e}" 656 return True, "Success"
Remove a plugin's virtual environment.
659 def uninstall(self, debug: bool = False) -> SuccessTuple: 660 """ 661 Remove a plugin, its virtual environment, and archive file. 662 """ 663 from meerschaum.utils.packages import reload_meerschaum 664 from meerschaum.plugins import sync_plugins_symlinks 665 from meerschaum.utils.warnings import warn, info 666 warnings_thrown_count: int = 0 667 max_warnings: int = 3 668 669 if not self.is_installed(): 670 info( 671 f"Plugin '{self.name}' doesn't seem to be installed.\n " 672 + "Checking for artifacts...", 673 stack = False, 674 ) 675 else: 676 real_path = pathlib.Path(os.path.realpath(self.__file__)) 677 try: 678 if real_path.name == '__init__.py': 679 shutil.rmtree(real_path.parent) 680 else: 681 real_path.unlink() 682 except Exception as e: 683 warn(f"Could not remove source files for plugin '{self.name}':\n{e}", stack=False) 684 warnings_thrown_count += 1 685 else: 686 info(f"Removed source files for plugin '{self.name}'.") 687 688 if self.venv_path.exists(): 689 success, msg = self.remove_venv(debug=debug) 690 if not success: 691 warn(msg, stack=False) 692 warnings_thrown_count += 1 693 else: 694 info(f"Removed virtual environment from plugin '{self.name}'.") 695 696 success = warnings_thrown_count < max_warnings 697 try: 698 from meerschaum.plugins._origins import remove_plugin_origin 699 remove_plugin_origin(self.name, debug=debug) 700 except Exception: 701 pass 702 sync_plugins_symlinks(debug=debug) 703 self.deactivate_venv(force=True, debug=debug) 704 reload_meerschaum(debug=debug) 705 return success, ( 706 f"Successfully uninstalled plugin '{self}'." if success 707 else f"Failed to uninstall plugin '{self}'." 708 )
Remove a plugin, its virtual environment, and archive file.
711 def setup(self, *args: str, debug: bool = False, **kw: Any) -> Union[SuccessTuple, bool]: 712 """ 713 If exists, run the plugin's `setup()` function. 714 715 Parameters 716 ---------- 717 *args: str 718 The positional arguments passed to the `setup()` function. 719 720 debug: bool, default False 721 Verbosity toggle. 722 723 **kw: Any 724 The keyword arguments passed to the `setup()` function. 725 726 Returns 727 ------- 728 A `SuccessTuple` or `bool` indicating success. 729 730 """ 731 from meerschaum.utils.debug import dprint 732 import inspect 733 _setup = None 734 for name, fp in inspect.getmembers(self.module): 735 if name == 'setup' and inspect.isfunction(fp): 736 _setup = fp 737 break 738 739 ### assume success if no setup() is found (not necessary) 740 if _setup is None: 741 return True 742 743 sig = inspect.signature(_setup) 744 has_debug, has_kw = ('debug' in sig.parameters), False 745 for k, v in sig.parameters.items(): 746 if '**' in str(v): 747 has_kw = True 748 break 749 750 _kw = {} 751 if has_kw: 752 _kw.update(kw) 753 if has_debug: 754 _kw['debug'] = debug 755 756 if debug: 757 dprint(f"Running setup for plugin '{self}'...") 758 from meerschaum.utils.venv import Venv 759 try: 760 with Venv(self, debug=debug): 761 return_tuple = _setup(*args, **_kw) 762 except Exception as e: 763 return False, str(e) 764 765 if isinstance(return_tuple, tuple): 766 return return_tuple 767 if isinstance(return_tuple, bool): 768 return return_tuple, f"Setup for Plugin '{self.name}' did not return a message." 769 if return_tuple is None: 770 return False, f"Setup for Plugin '{self.name}' returned None." 771 return False, f"Unknown return value from setup for Plugin '{self.name}': {return_tuple}"
If exists, run the plugin's setup() function.
Parameters
- *args (str):
The positional arguments passed to the
setup()function. - debug (bool, default False): Verbosity toggle.
- **kw (Any):
The keyword arguments passed to the
setup()function.
Returns
- A
SuccessTupleorboolindicating success.
774 def get_dependencies( 775 self, 776 debug: bool = False, 777 ) -> List[str]: 778 """ 779 If the Plugin has specified dependencies in a list called `required`, return the list. 780 781 **NOTE:** Dependecies which start with `'plugin:'` are Meerschaum plugins, not pip packages. 782 Meerschaum plugins may also specify connector keys for a repo after `'@'`. 783 784 Parameters 785 ---------- 786 debug: bool, default False 787 Verbosity toggle. 788 789 Returns 790 ------- 791 A list of required packages and plugins (str). 792 793 """ 794 if '_required' in self.__dict__: 795 return self._required 796 797 ### If the plugin has not yet been imported, 798 ### infer the dependencies from the source text. 799 ### This is not super robust, and it doesn't feel right 800 ### having multiple versions of the logic. 801 ### This is necessary when determining the activation order 802 ### without having import the module. 803 ### For consistency's sake, the module-less method does not cache the requirements. 804 if self.__dict__.get('_module', None) is None: 805 file_path = self.__file__ 806 if file_path is None: 807 return [] 808 with open(file_path, 'r', encoding='utf-8') as f: 809 text = f.read() 810 811 if 'required' not in text: 812 return [] 813 814 ### This has some limitations: 815 ### It relies on `required` being manually declared. 816 ### We lose the ability to dynamically alter the `required` list, 817 ### which is why we've kept the module-reliant method below. 818 import ast, re 819 ### NOTE: This technically would break 820 ### if `required` was the very first line of the file. 821 req_start_match = re.search(r'\nrequired(:\s*)?.*=', text) 822 if not req_start_match: 823 return [] 824 req_start = req_start_match.start() 825 equals_sign = req_start + text[req_start:].find('=') 826 827 ### Dependencies may have brackets within the strings, so push back the index. 828 first_opening_brace = equals_sign + 1 + text[equals_sign:].find('[') 829 if first_opening_brace == -1: 830 return [] 831 832 next_closing_brace = equals_sign + 1 + text[equals_sign:].find(']') 833 if next_closing_brace == -1: 834 return [] 835 836 start_ix = first_opening_brace + 1 837 end_ix = next_closing_brace 838 839 num_braces = 0 840 while True: 841 if '[' not in text[start_ix:end_ix]: 842 break 843 num_braces += 1 844 start_ix = end_ix 845 end_ix += text[end_ix + 1:].find(']') + 1 846 847 req_end = end_ix + 1 848 req_text = ( 849 text[(first_opening_brace-1):req_end] 850 .lstrip() 851 .replace('=', '', 1) 852 .lstrip() 853 .rstrip() 854 ) 855 try: 856 required = ast.literal_eval(req_text) 857 except Exception as e: 858 warn( 859 f"Unable to determine requirements for plugin '{self.name}' " 860 + "without importing the module.\n" 861 + " This may be due to dynamically setting the global `required` list.\n" 862 + f" {e}" 863 ) 864 return [] 865 return required 866 867 import inspect 868 self.activate_venv(dependencies=False, debug=debug) 869 required = [] 870 for name, val in inspect.getmembers(self.module): 871 if name == 'required': 872 required = val 873 break 874 self._required = required 875 self.deactivate_venv(dependencies=False, debug=debug) 876 return required
If the Plugin has specified dependencies in a list called required, return the list.
NOTE: Dependecies which start with 'plugin:' are Meerschaum plugins, not pip packages.
Meerschaum plugins may also specify connector keys for a repo after '@'.
Parameters
- debug (bool, default False): Verbosity toggle.
Returns
- A list of required packages and plugins (str).
879 def get_required_plugins(self, debug: bool=False) -> List[mrsm.plugins.Plugin]: 880 """ 881 Return a list of required Plugin objects. 882 """ 883 from meerschaum.utils.warnings import warn 884 from meerschaum.config import get_config 885 from meerschaum._internal.static import STATIC_CONFIG 886 from meerschaum.connectors.parse import is_valid_connector_keys 887 plugins = [] 888 _deps = self.get_dependencies(debug=debug) 889 sep = STATIC_CONFIG['plugins']['repo_separator'] 890 plugin_names = [ 891 _d[len('plugin:'):] for _d in _deps 892 if _d.startswith('plugin:') and len(_d) > len('plugin:') 893 ] 894 default_repo_keys = get_config('meerschaum', 'repository') 895 skipped_repo_keys = set() 896 897 for _plugin_name in plugin_names: 898 if sep in _plugin_name: 899 try: 900 _plugin_name, _repo_keys = _plugin_name.split(sep) 901 except Exception: 902 _repo_keys = default_repo_keys 903 warn( 904 f"Invalid repo keys for required plugin '{_plugin_name}'.\n " 905 + f"Will try to use '{_repo_keys}' instead.", 906 stack = False, 907 ) 908 else: 909 _repo_keys = default_repo_keys 910 911 if _repo_keys in skipped_repo_keys: 912 continue 913 914 if not is_valid_connector_keys(_repo_keys): 915 warn( 916 f"Invalid connector '{_repo_keys}'.\n" 917 f" Skipping required plugins from repository '{_repo_keys}'", 918 stack=False, 919 ) 920 continue 921 922 plugins.append(Plugin(_plugin_name, repo=_repo_keys)) 923 924 return plugins
Return a list of required Plugin objects.
927 def get_required_packages(self, debug: bool=False) -> List[str]: 928 """ 929 Return the required package names (excluding plugins). 930 """ 931 _deps = self.get_dependencies(debug=debug) 932 return [_d for _d in _deps if not _d.startswith('plugin:')]
Return the required package names (excluding plugins).
935 def activate_venv( 936 self, 937 dependencies: bool = True, 938 init_if_not_exists: bool = True, 939 debug: bool = False, 940 **kw 941 ) -> bool: 942 """ 943 Activate the virtual environments for the plugin and its dependencies. 944 945 Parameters 946 ---------- 947 dependencies: bool, default True 948 If `True`, activate the virtual environments for required plugins. 949 950 Returns 951 ------- 952 A bool indicating success. 953 """ 954 import meerschaum.config.paths as paths 955 from meerschaum.utils.venv import venv_target_path 956 from meerschaum.utils.packages import activate_venv 957 from meerschaum.utils.misc import make_symlink, is_symlink 958 959 if dependencies: 960 for plugin in self.get_required_plugins(debug=debug): 961 plugin.activate_venv(debug=debug, init_if_not_exists=init_if_not_exists, **kw) 962 963 vtp = venv_target_path(self.name, debug=debug, allow_nonexistent=True) 964 venv_meerschaum_path = vtp / 'meerschaum' 965 966 try: 967 success, msg = True, "Success" 968 if is_symlink(venv_meerschaum_path): 969 if pathlib.Path(os.path.realpath(venv_meerschaum_path)) != paths.PACKAGE_ROOT_PATH: 970 venv_meerschaum_path.unlink() 971 success, msg = make_symlink(venv_meerschaum_path, paths.PACKAGE_ROOT_PATH) 972 except Exception as e: 973 success, msg = False, str(e) 974 if not success: 975 warn( 976 f"Unable to create symlink {venv_meerschaum_path} to {paths.PACKAGE_ROOT_PATH}:\n" 977 f"{msg}" 978 ) 979 980 return activate_venv(self.name, init_if_not_exists=init_if_not_exists, debug=debug, **kw)
Activate the virtual environments for the plugin and its dependencies.
Parameters
- dependencies (bool, default True):
If
True, activate the virtual environments for required plugins.
Returns
- A bool indicating success.
983 def deactivate_venv(self, dependencies: bool=True, debug: bool = False, **kw) -> bool: 984 """ 985 Deactivate the virtual environments for the plugin and its dependencies. 986 987 Parameters 988 ---------- 989 dependencies: bool, default True 990 If `True`, deactivate the virtual environments for required plugins. 991 992 Returns 993 ------- 994 A bool indicating success. 995 """ 996 from meerschaum.utils.packages import deactivate_venv 997 success = deactivate_venv(self.name, debug=debug, **kw) 998 if dependencies: 999 for plugin in self.get_required_plugins(debug=debug): 1000 plugin.deactivate_venv(debug=debug, **kw) 1001 return success
Deactivate the virtual environments for the plugin and its dependencies.
Parameters
- dependencies (bool, default True):
If
True, deactivate the virtual environments for required plugins.
Returns
- A bool indicating success.
1004 def install_dependencies( 1005 self, 1006 force: bool = False, 1007 debug: bool = False, 1008 ) -> bool: 1009 """ 1010 If specified, install dependencies. 1011 1012 **NOTE:** Dependencies that start with `'plugin:'` will be installed as 1013 Meerschaum plugins from the same repository as this Plugin. 1014 To install from a different repository, add the repo keys after `'@'` 1015 (e.g. `'plugin:foo@api:bar'`). 1016 1017 Parameters 1018 ---------- 1019 force: bool, default False 1020 If `True`, continue with the installation, even if some 1021 required packages fail to install. 1022 1023 debug: bool, default False 1024 Verbosity toggle. 1025 1026 Returns 1027 ------- 1028 A bool indicating success. 1029 """ 1030 from meerschaum.utils.packages import pip_install, venv_contains_package 1031 from meerschaum.utils.warnings import warn, info 1032 _deps = self.get_dependencies(debug=debug) 1033 if not _deps and self.requirements_file_path is None: 1034 return True 1035 1036 plugins = self.get_required_plugins(debug=debug) 1037 for _plugin in plugins: 1038 if _plugin.name == self.name: 1039 warn(f"Plugin '{self.name}' cannot depend on itself! Skipping...", stack=False) 1040 continue 1041 _success, _msg = _plugin.repo_connector.install_plugin( 1042 _plugin.name, debug=debug, force=force 1043 ) 1044 if not _success: 1045 warn( 1046 f"Failed to install required plugin '{_plugin}' from '{_plugin.repo_connector}'" 1047 + f" for plugin '{self.name}':\n" + _msg, 1048 stack = False, 1049 ) 1050 if not force: 1051 warn( 1052 "Try installing with the `--force` flag to continue anyway.", 1053 stack = False, 1054 ) 1055 return False 1056 info( 1057 "Continuing with installation despite the failure " 1058 + "(careful, things might be broken!)...", 1059 icon = False 1060 ) 1061 1062 1063 ### First step: parse `requirements.txt` if it exists. 1064 if self.requirements_file_path is not None: 1065 if not pip_install( 1066 requirements_file_path=self.requirements_file_path, 1067 venv=self.name, debug=debug 1068 ): 1069 warn( 1070 f"Failed to resolve 'requirements.txt' for plugin '{self.name}'.", 1071 stack = False, 1072 ) 1073 if not force: 1074 warn( 1075 "Try installing with `--force` to continue anyway.", 1076 stack = False, 1077 ) 1078 return False 1079 info( 1080 "Continuing with installation despite the failure " 1081 + "(careful, things might be broken!)...", 1082 icon = False 1083 ) 1084 1085 1086 ### Don't reinstall packages that are already included in required plugins. 1087 packages = [] 1088 _packages = self.get_required_packages(debug=debug) 1089 accounted_for_packages = set() 1090 for package_name in _packages: 1091 for plugin in plugins: 1092 if venv_contains_package(package_name, plugin.name): 1093 accounted_for_packages.add(package_name) 1094 break 1095 packages = [pkg for pkg in _packages if pkg not in accounted_for_packages] 1096 1097 ### Attempt pip packages installation. 1098 if packages: 1099 for package in packages: 1100 if not pip_install(package, venv=self.name, debug=debug): 1101 warn( 1102 f"Failed to install required package '{package}'" 1103 + f" for plugin '{self.name}'.", 1104 stack = False, 1105 ) 1106 if not force: 1107 warn( 1108 "Try installing with `--force` to continue anyway.", 1109 stack = False, 1110 ) 1111 return False 1112 info( 1113 "Continuing with installation despite the failure " 1114 + "(careful, things might be broken!)...", 1115 icon = False 1116 ) 1117 return True
If specified, install dependencies.
NOTE: Dependencies that start with 'plugin:' will be installed as
Meerschaum plugins from the same repository as this Plugin.
To install from a different repository, add the repo keys after '@'
(e.g. 'plugin:foo@api:bar').
Parameters
- force (bool, default False):
If
True, continue with the installation, even if some required packages fail to install. - debug (bool, default False): Verbosity toggle.
Returns
- A bool indicating success.
1120 @property 1121 def full_name(self) -> str: 1122 """ 1123 Include the repo keys with the plugin's name. 1124 """ 1125 from meerschaum._internal.static import STATIC_CONFIG 1126 sep = STATIC_CONFIG['plugins']['repo_separator'] 1127 return self.name + sep + str(self.repo_connector)
Include the repo keys with the plugin's name.
19class Venv: 20 """ 21 Manage a virtual enviroment's activation status. 22 23 Examples 24 -------- 25 >>> from meerschaum.plugins import Plugin 26 >>> with Venv('mrsm') as venv: 27 ... import pandas 28 >>> with Venv(Plugin('noaa')) as venv: 29 ... import requests 30 >>> venv = Venv('mrsm') 31 >>> venv.activate() 32 True 33 >>> venv.deactivate() 34 True 35 >>> 36 """ 37 38 def __init__( 39 self, 40 venv: Union[str, 'mrsm.core.Plugin', None] = 'mrsm', 41 init_if_not_exists: bool = True, 42 debug: bool = False, 43 ) -> None: 44 from meerschaum.utils.venv import activate_venv, deactivate_venv, active_venvs 45 ### For some weird threading issue, 46 ### we can't use `isinstance` here. 47 if '_Plugin' in str(type(venv)): 48 self._venv = venv.name 49 self._activate = venv.activate_venv 50 self._deactivate = venv.deactivate_venv 51 self._kwargs = {} 52 else: 53 self._venv = venv 54 self._activate = activate_venv 55 self._deactivate = deactivate_venv 56 self._kwargs = {'venv': venv} 57 self._debug = debug 58 self._init_if_not_exists = init_if_not_exists 59 ### In case someone calls `deactivate()` before `activate()`. 60 self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs) 61 62 63 def activate(self, debug: bool = False) -> bool: 64 """ 65 Activate this virtual environment. 66 If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments 67 will also be activated. 68 """ 69 from meerschaum.utils.venv import active_venvs, init_venv 70 self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs) 71 try: 72 return self._activate( 73 debug=(debug or self._debug), 74 init_if_not_exists=self._init_if_not_exists, 75 **self._kwargs 76 ) 77 except OSError as e: 78 if self._init_if_not_exists: 79 if not init_venv(self._venv, force=True): 80 raise e 81 return self._activate( 82 debug=(debug or self._debug), 83 init_if_not_exists=self._init_if_not_exists, 84 **self._kwargs 85 ) 86 87 88 def deactivate(self, debug: bool = False) -> bool: 89 """ 90 Deactivate this virtual environment. 91 If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments 92 will also be deactivated. 93 """ 94 return self._deactivate(debug=(debug or self._debug), **self._kwargs) 95 96 97 @property 98 def target_path(self) -> pathlib.Path: 99 """ 100 Return the target site-packages path for this virtual environment. 101 A `meerschaum.utils.venv.Venv` may have one virtual environment per minor Python version 102 (e.g. Python 3.10 and Python 3.7). 103 """ 104 from meerschaum.utils.venv import venv_target_path 105 return venv_target_path(venv=self._venv, allow_nonexistent=True, debug=self._debug) 106 107 108 @property 109 def root_path(self) -> pathlib.Path: 110 """ 111 Return the top-level path for this virtual environment. 112 """ 113 import meerschaum.config.paths as paths 114 if self._venv is None: 115 return self.target_path.parent 116 return paths.VIRTENV_RESOURCES_PATH / self._venv 117 118 119 def __enter__(self) -> None: 120 self.activate(debug=self._debug) 121 122 123 def __exit__(self, exc_type, exc_value, exc_traceback) -> None: 124 self.deactivate(debug=self._debug) 125 126 127 def __str__(self) -> str: 128 quote = "'" if self._venv is not None else "" 129 return "Venv(" + quote + str(self._venv) + quote + ")" 130 131 132 def __repr__(self) -> str: 133 return self.__str__()
Manage a virtual enviroment's activation status.
Examples
>>> from meerschaum.plugins import Plugin
>>> with Venv('mrsm') as venv:
... import pandas
>>> with Venv(Plugin('noaa')) as venv:
... import requests
>>> venv = Venv('mrsm')
>>> venv.activate()
True
>>> venv.deactivate()
True
>>>
38 def __init__( 39 self, 40 venv: Union[str, 'mrsm.core.Plugin', None] = 'mrsm', 41 init_if_not_exists: bool = True, 42 debug: bool = False, 43 ) -> None: 44 from meerschaum.utils.venv import activate_venv, deactivate_venv, active_venvs 45 ### For some weird threading issue, 46 ### we can't use `isinstance` here. 47 if '_Plugin' in str(type(venv)): 48 self._venv = venv.name 49 self._activate = venv.activate_venv 50 self._deactivate = venv.deactivate_venv 51 self._kwargs = {} 52 else: 53 self._venv = venv 54 self._activate = activate_venv 55 self._deactivate = deactivate_venv 56 self._kwargs = {'venv': venv} 57 self._debug = debug 58 self._init_if_not_exists = init_if_not_exists 59 ### In case someone calls `deactivate()` before `activate()`. 60 self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs)
63 def activate(self, debug: bool = False) -> bool: 64 """ 65 Activate this virtual environment. 66 If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments 67 will also be activated. 68 """ 69 from meerschaum.utils.venv import active_venvs, init_venv 70 self._kwargs['previously_active_venvs'] = copy.deepcopy(active_venvs) 71 try: 72 return self._activate( 73 debug=(debug or self._debug), 74 init_if_not_exists=self._init_if_not_exists, 75 **self._kwargs 76 ) 77 except OSError as e: 78 if self._init_if_not_exists: 79 if not init_venv(self._venv, force=True): 80 raise e 81 return self._activate( 82 debug=(debug or self._debug), 83 init_if_not_exists=self._init_if_not_exists, 84 **self._kwargs 85 )
Activate this virtual environment.
If a meerschaum.plugins.Plugin was provided, its dependent virtual environments
will also be activated.
88 def deactivate(self, debug: bool = False) -> bool: 89 """ 90 Deactivate this virtual environment. 91 If a `meerschaum.plugins.Plugin` was provided, its dependent virtual environments 92 will also be deactivated. 93 """ 94 return self._deactivate(debug=(debug or self._debug), **self._kwargs)
Deactivate this virtual environment.
If a meerschaum.plugins.Plugin was provided, its dependent virtual environments
will also be deactivated.
97 @property 98 def target_path(self) -> pathlib.Path: 99 """ 100 Return the target site-packages path for this virtual environment. 101 A `meerschaum.utils.venv.Venv` may have one virtual environment per minor Python version 102 (e.g. Python 3.10 and Python 3.7). 103 """ 104 from meerschaum.utils.venv import venv_target_path 105 return venv_target_path(venv=self._venv, allow_nonexistent=True, debug=self._debug)
Return the target site-packages path for this virtual environment.
A meerschaum.utils.venv.Venv may have one virtual environment per minor Python version
(e.g. Python 3.10 and Python 3.7).
108 @property 109 def root_path(self) -> pathlib.Path: 110 """ 111 Return the top-level path for this virtual environment. 112 """ 113 import meerschaum.config.paths as paths 114 if self._venv is None: 115 return self.target_path.parent 116 return paths.VIRTENV_RESOURCES_PATH / self._venv
Return the top-level path for this virtual environment.
70class Job: 71 """ 72 Manage a `meerschaum.utils.daemon.Daemon`, locally or remotely via the API. 73 """ 74 75 def __init__( 76 self, 77 name: str, 78 sysargs: Union[List[str], str, None] = None, 79 env: Optional[Dict[str, str]] = None, 80 executor_keys: Optional[str] = None, 81 delete_after_completion: bool = False, 82 refresh_seconds: Union[int, float, None] = None, 83 _properties: Optional[Dict[str, Any]] = None, 84 _rotating_log=None, 85 _stdin_file=None, 86 _status_hook: Optional[Callable[[], str]] = None, 87 _result_hook: Optional[Callable[[], SuccessTuple]] = None, 88 _externally_managed: bool = False, 89 ): 90 """ 91 Create a new job to manage a `meerschaum.utils.daemon.Daemon`. 92 93 Parameters 94 ---------- 95 name: str 96 The name of the job to be created. 97 This will also be used as the Daemon ID. 98 99 sysargs: Union[List[str], str, None], default None 100 The sysargs of the command to be executed, e.g. 'start api'. 101 102 env: Optional[Dict[str, str]], default None 103 If provided, set these environment variables in the job's process. 104 105 executor_keys: Optional[str], default None 106 If provided, execute the job remotely on an API instance, e.g. 'api:main'. 107 108 delete_after_completion: bool, default False 109 If `True`, delete this job when it has finished executing. 110 111 refresh_seconds: Union[int, float, None], default None 112 The number of seconds to sleep between refreshes. 113 Defaults to the configured value `system.cli.refresh_seconds`. 114 115 _properties: Optional[Dict[str, Any]], default None 116 If provided, use this to patch the daemon's properties. 117 """ 118 from meerschaum.utils.daemon import Daemon 119 for char in BANNED_CHARS: 120 if char in name: 121 raise ValueError(f"Invalid name: ({char}) is not allowed.") 122 123 if isinstance(sysargs, str): 124 sysargs = shlex.split(sysargs) 125 126 and_key = STATIC_CONFIG['system']['arguments']['and_key'] 127 escaped_and_key = STATIC_CONFIG['system']['arguments']['escaped_and_key'] 128 if sysargs: 129 sysargs = [ 130 (arg if arg != escaped_and_key else and_key) 131 for arg in sysargs 132 ] 133 134 ### NOTE: 'local' and 'systemd' executors are being coalesced. 135 if executor_keys is None: 136 from meerschaum.jobs import get_executor_keys_from_context 137 executor_keys = get_executor_keys_from_context() 138 139 self.executor_keys = executor_keys 140 self.name = name 141 self.refresh_seconds = ( 142 refresh_seconds 143 if refresh_seconds is not None 144 else mrsm.get_config('system', 'cli', 'refresh_seconds') 145 ) 146 try: 147 self._daemon = ( 148 Daemon(daemon_id=name) 149 if executor_keys == 'local' 150 else None 151 ) 152 except Exception: 153 self._daemon = None 154 155 ### Handle any injected dependencies. 156 if _rotating_log is not None: 157 self._rotating_log = _rotating_log 158 if self._daemon is not None: 159 self._daemon._rotating_log = _rotating_log 160 161 if _stdin_file is not None: 162 self._stdin_file = _stdin_file 163 if self._daemon is not None: 164 self._daemon._stdin_file = _stdin_file 165 self._daemon._blocking_stdin_file_path = _stdin_file.blocking_file_path 166 167 if _status_hook is not None: 168 self._status_hook = _status_hook 169 170 if _result_hook is not None: 171 self._result_hook = _result_hook 172 173 self._externally_managed = _externally_managed 174 self._properties_patch = _properties or {} 175 if _externally_managed: 176 self._properties_patch.update({'externally_managed': _externally_managed}) 177 178 if env: 179 self._properties_patch.update({'env': env}) 180 181 if delete_after_completion: 182 self._properties_patch.update({'delete_after_completion': delete_after_completion}) 183 184 daemon_sysargs = ( 185 self._daemon.properties.get('target', {}).get('args', [None])[0] 186 if self._daemon is not None 187 else None 188 ) 189 190 if daemon_sysargs and sysargs and daemon_sysargs != sysargs: 191 warn("Given sysargs differ from existing sysargs.") 192 193 self._sysargs = [ 194 arg 195 for arg in (daemon_sysargs or sysargs or []) 196 if arg not in ('-d', '--daemon') 197 ] 198 for restart_flag in RESTART_FLAGS: 199 if restart_flag in self._sysargs: 200 self._properties_patch.update({'restart': True}) 201 break 202 203 @staticmethod 204 def from_pid(pid: int, executor_keys: Optional[str] = None) -> Job: 205 """ 206 Build a `Job` from the PID of a running Meerschaum process. 207 208 Parameters 209 ---------- 210 pid: int 211 The PID of the process. 212 213 executor_keys: Optional[str], default None 214 The executor keys to assign to the job. 215 """ 216 psutil = mrsm.attempt_import('psutil') 217 try: 218 process = psutil.Process(pid) 219 except psutil.NoSuchProcess as e: 220 warn(f"Process with PID {pid} does not exist.", stack=False) 221 raise e 222 223 command_args = process.cmdline() 224 is_daemon = command_args[1] == '-c' 225 226 if is_daemon: 227 daemon_id = command_args[-1].split('daemon_id=')[-1].split(')')[0].replace("'", '') 228 root_dir = process.environ().get(STATIC_CONFIG['environment']['root'], None) 229 if root_dir is None: 230 root_dir = paths.ROOT_DIR_PATH 231 else: 232 root_dir = pathlib.Path(root_dir) 233 jobs_dir = root_dir / paths.DAEMON_RESOURCES_PATH.name 234 daemon_dir = jobs_dir / daemon_id 235 pid_file = daemon_dir / 'process.pid' 236 237 if pid_file.exists(): 238 with open(pid_file, 'r', encoding='utf-8') as f: 239 daemon_pid = int(f.read()) 240 241 if pid != daemon_pid: 242 raise EnvironmentError(f"Differing PIDs: {pid=}, {daemon_pid=}") 243 else: 244 raise EnvironmentError(f"Is job '{daemon_id}' running?") 245 246 return Job(daemon_id, executor_keys=executor_keys) 247 248 from meerschaum._internal.arguments._parse_arguments import parse_arguments 249 from meerschaum.utils.daemon import get_new_daemon_name 250 251 mrsm_ix = 0 252 for i, arg in enumerate(command_args): 253 if 'mrsm' in arg or 'meerschaum' in arg.lower(): 254 mrsm_ix = i 255 break 256 257 sysargs = command_args[mrsm_ix+1:] 258 kwargs = parse_arguments(sysargs) 259 name = kwargs.get('name', get_new_daemon_name()) 260 return Job(name, sysargs, executor_keys=executor_keys) 261 262 def start(self, debug: bool = False) -> SuccessTuple: 263 """ 264 Start the job's daemon. 265 """ 266 if self.executor is not None: 267 if not self.exists(debug=debug): 268 return self.executor.create_job( 269 self.name, 270 self.sysargs, 271 properties=self.daemon.properties, 272 debug=debug, 273 ) 274 return self.executor.start_job(self.name, debug=debug) 275 276 if self.is_running(): 277 return True, f"{self} is already running." 278 279 success, msg = self.daemon.run( 280 keep_daemon_output=(not self.delete_after_completion), 281 allow_dirty_run=True, 282 ) 283 if not success: 284 return success, msg 285 286 return success, f"Started {self}." 287 288 def stop( 289 self, 290 timeout_seconds: Union[int, float, None] = None, 291 debug: bool = False, 292 ) -> SuccessTuple: 293 """ 294 Stop the job's daemon. 295 """ 296 if self.executor is not None: 297 return self.executor.stop_job(self.name, debug=debug) 298 299 if self.daemon.status == 'stopped': 300 if not self.restart: 301 return True, f"{self} is not running." 302 elif self.stop_time is not None: 303 return True, f"{self} will not restart until manually started." 304 305 quit_success, quit_msg = self.daemon.quit(timeout=timeout_seconds) 306 if quit_success: 307 return quit_success, f"Stopped {self}." 308 309 warn( 310 f"Failed to gracefully quit {self}: {quit_msg}", 311 stack=False, 312 ) 313 kill_success, kill_msg = self.daemon.kill(timeout=timeout_seconds) 314 if not kill_success: 315 return kill_success, kill_msg 316 317 return kill_success, f"Killed {self}." 318 319 def pause( 320 self, 321 timeout_seconds: Union[int, float, None] = None, 322 debug: bool = False, 323 ) -> SuccessTuple: 324 """ 325 Pause the job's daemon. 326 """ 327 if self.executor is not None: 328 return self.executor.pause_job(self.name, debug=debug) 329 330 pause_success, pause_msg = self.daemon.pause(timeout=timeout_seconds) 331 if not pause_success: 332 return pause_success, pause_msg 333 334 return pause_success, f"Paused {self}." 335 336 def delete(self, debug: bool = False) -> SuccessTuple: 337 """ 338 Delete the job and its daemon. 339 """ 340 if self.executor is not None: 341 return self.executor.delete_job(self.name, debug=debug) 342 343 if self.is_running(): 344 stop_success, stop_msg = self.stop() 345 if not stop_success: 346 return stop_success, stop_msg 347 348 cleanup_success, cleanup_msg = self.daemon.cleanup() 349 if not cleanup_success: 350 return cleanup_success, cleanup_msg 351 352 _ = self.daemon._properties.pop('result', None) 353 return cleanup_success, f"Deleted {self}." 354 355 def is_running(self) -> bool: 356 """ 357 Determine whether the job's daemon is running. 358 """ 359 return self.status == 'running' 360 361 def exists(self, debug: bool = False) -> bool: 362 """ 363 Determine whether the job exists. 364 """ 365 if self.executor is not None: 366 return self.executor.get_job_exists(self.name, debug=debug) 367 368 return self.daemon.path.exists() 369 370 def get_logs(self) -> Union[str, None]: 371 """ 372 Return the output text of the job's daemon. 373 """ 374 if self.executor is not None: 375 return self.executor.get_logs(self.name) 376 377 return self.daemon.log_text 378 379 def monitor_logs( 380 self, 381 callback_function: Callable[[str], None] = _default_stdout_callback, 382 input_callback_function: Optional[Callable[[], str]] = None, 383 stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None, 384 stop_event: Optional[asyncio.Event] = None, 385 stop_on_exit: bool = False, 386 strip_timestamps: bool = False, 387 accept_input: bool = True, 388 debug: bool = False, 389 _logs_path: Optional[pathlib.Path] = None, 390 _log=None, 391 _stdin_file=None, 392 _wait_if_stopped: bool = True, 393 ): 394 """ 395 Monitor the job's log files and execute a callback on new lines. 396 397 Parameters 398 ---------- 399 callback_function: Callable[[str], None], default partial(print, end='') 400 The callback to execute as new data comes in. 401 Defaults to printing the output directly to `stdout`. 402 403 input_callback_function: Optional[Callable[[], str]], default None 404 If provided, execute this callback when the daemon is blocking on stdin. 405 Defaults to `sys.stdin.readline()`. 406 407 stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None 408 If provided, execute this callback when the daemon stops. 409 The job's SuccessTuple will be passed to the callback. 410 411 stop_event: Optional[asyncio.Event], default None 412 If provided, stop monitoring when this event is set. 413 You may instead raise `meerschaum.jobs.StopMonitoringLogs` 414 from within `callback_function` to stop monitoring. 415 416 stop_on_exit: bool, default False 417 If `True`, stop monitoring when the job stops. 418 419 strip_timestamps: bool, default False 420 If `True`, remove leading timestamps from lines. 421 422 accept_input: bool, default True 423 If `True`, accept input when the daemon blocks on stdin. 424 """ 425 if self.executor is not None: 426 self.executor.monitor_logs( 427 self.name, 428 callback_function, 429 input_callback_function=input_callback_function, 430 stop_callback_function=stop_callback_function, 431 stop_on_exit=stop_on_exit, 432 accept_input=accept_input, 433 strip_timestamps=strip_timestamps, 434 debug=debug, 435 ) 436 return 437 438 monitor_logs_coroutine = self.monitor_logs_async( 439 callback_function=callback_function, 440 input_callback_function=input_callback_function, 441 stop_callback_function=stop_callback_function, 442 stop_event=stop_event, 443 stop_on_exit=stop_on_exit, 444 strip_timestamps=strip_timestamps, 445 accept_input=accept_input, 446 debug=debug, 447 _logs_path=_logs_path, 448 _log=_log, 449 _stdin_file=_stdin_file, 450 _wait_if_stopped=_wait_if_stopped, 451 ) 452 return asyncio.run(monitor_logs_coroutine) 453 454 async def monitor_logs_async( 455 self, 456 callback_function: Callable[[str], None] = _default_stdout_callback, 457 input_callback_function: Optional[Callable[[], str]] = None, 458 stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None, 459 stop_event: Optional[asyncio.Event] = None, 460 stop_on_exit: bool = False, 461 strip_timestamps: bool = False, 462 accept_input: bool = True, 463 debug: bool = False, 464 _logs_path: Optional[pathlib.Path] = None, 465 _log=None, 466 _stdin_file=None, 467 _wait_if_stopped: bool = True, 468 ): 469 """ 470 Monitor the job's log files and await a callback on new lines. 471 472 Parameters 473 ---------- 474 callback_function: Callable[[str], None], default _default_stdout_callback 475 The callback to execute as new data comes in. 476 Defaults to printing the output directly to `stdout`. 477 478 input_callback_function: Optional[Callable[[], str]], default None 479 If provided, execute this callback when the daemon is blocking on stdin. 480 Defaults to `sys.stdin.readline()`. 481 482 stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None 483 If provided, execute this callback when the daemon stops. 484 The job's SuccessTuple will be passed to the callback. 485 486 stop_event: Optional[asyncio.Event], default None 487 If provided, stop monitoring when this event is set. 488 You may instead raise `meerschaum.jobs.StopMonitoringLogs` 489 from within `callback_function` to stop monitoring. 490 491 stop_on_exit: bool, default False 492 If `True`, stop monitoring when the job stops. 493 494 strip_timestamps: bool, default False 495 If `True`, remove leading timestamps from lines. 496 497 accept_input: bool, default True 498 If `True`, accept input when the daemon blocks on stdin. 499 """ 500 from meerschaum.utils.prompt import prompt 501 502 def default_input_callback_function(): 503 prompt_kwargs = self.get_prompt_kwargs(debug=debug) 504 if prompt_kwargs: 505 answer = prompt(**prompt_kwargs) 506 return answer + '\n' 507 return sys.stdin.readline() 508 509 if input_callback_function is None: 510 input_callback_function = default_input_callback_function 511 512 if self.executor is not None: 513 await self.executor.monitor_logs_async( 514 self.name, 515 callback_function, 516 input_callback_function=input_callback_function, 517 stop_callback_function=stop_callback_function, 518 stop_on_exit=stop_on_exit, 519 strip_timestamps=strip_timestamps, 520 accept_input=accept_input, 521 debug=debug, 522 ) 523 return 524 525 from meerschaum.utils.formatting._jobs import strip_timestamp_from_line 526 527 events = { 528 'user': stop_event, 529 'stopped': asyncio.Event(), 530 'stop_token': asyncio.Event(), 531 'stop_exception': asyncio.Event(), 532 'stopped_timeout': asyncio.Event(), 533 } 534 combined_event = asyncio.Event() 535 emitted_text = False 536 stdin_file = _stdin_file if _stdin_file is not None else self.daemon.stdin_file 537 538 async def check_job_status(): 539 if not stop_on_exit: 540 return 541 542 nonlocal emitted_text 543 544 sleep_time = 0.1 545 while sleep_time < 0.2: 546 if self.status == 'stopped': 547 if not emitted_text and _wait_if_stopped: 548 await asyncio.sleep(sleep_time) 549 sleep_time = round(sleep_time * 1.1, 3) 550 continue 551 552 if stop_callback_function is not None: 553 try: 554 if asyncio.iscoroutinefunction(stop_callback_function): 555 await stop_callback_function(self.result) 556 else: 557 stop_callback_function(self.result) 558 except asyncio.exceptions.CancelledError: 559 break 560 except Exception: 561 warn(traceback.format_exc()) 562 563 if stop_on_exit: 564 events['stopped'].set() 565 566 break 567 await asyncio.sleep(0.1) 568 569 events['stopped_timeout'].set() 570 571 async def check_blocking_on_input(): 572 while True: 573 if not emitted_text or not self.is_blocking_on_stdin(): 574 try: 575 await asyncio.sleep(self.refresh_seconds) 576 except asyncio.exceptions.CancelledError: 577 break 578 continue 579 580 if not self.is_running(): 581 break 582 583 await emit_latest_lines() 584 585 try: 586 print('', end='', flush=True) 587 if asyncio.iscoroutinefunction(input_callback_function): 588 data = await input_callback_function() 589 else: 590 loop = asyncio.get_running_loop() 591 data = await loop.run_in_executor(None, input_callback_function) 592 except KeyboardInterrupt: 593 break 594 # if not data.endswith('\n'): 595 # data += '\n' 596 597 stdin_file.write(data) 598 await asyncio.sleep(self.refresh_seconds) 599 600 async def combine_events(): 601 event_tasks = [ 602 asyncio.create_task(event.wait()) 603 for event in events.values() 604 if event is not None 605 ] 606 if not event_tasks: 607 return 608 609 try: 610 done, pending = await asyncio.wait( 611 event_tasks, 612 return_when=asyncio.FIRST_COMPLETED, 613 ) 614 for task in pending: 615 task.cancel() 616 except asyncio.exceptions.CancelledError: 617 pass 618 finally: 619 combined_event.set() 620 621 check_job_status_task = asyncio.create_task(check_job_status()) 622 check_blocking_on_input_task = asyncio.create_task(check_blocking_on_input()) 623 combine_events_task = asyncio.create_task(combine_events()) 624 625 log = _log if _log is not None else self.daemon.rotating_log 626 lines_to_show = ( 627 self.daemon.properties.get( 628 'logs', {} 629 ).get( 630 'lines_to_show', get_config('jobs', 'logs', 'lines_to_show') 631 ) 632 ) 633 634 async def emit_latest_lines(): 635 nonlocal emitted_text 636 nonlocal stop_event 637 lines = log.readlines() 638 for line in lines[(-1 * lines_to_show):]: 639 if stop_event is not None and stop_event.is_set(): 640 return 641 642 line_stripped_extra = strip_timestamp_from_line(line.strip()) 643 line_stripped = strip_timestamp_from_line(line) 644 645 if line_stripped_extra == STOP_TOKEN: 646 events['stop_token'].set() 647 return 648 649 if line_stripped_extra == CLEAR_TOKEN: 650 clear_screen(debug=debug) 651 continue 652 653 if line_stripped_extra == FLUSH_TOKEN.strip(): 654 line_stripped = '' 655 line = '' 656 657 if strip_timestamps: 658 line = line_stripped 659 660 try: 661 if asyncio.iscoroutinefunction(callback_function): 662 await callback_function(line) 663 else: 664 callback_function(line) 665 emitted_text = True 666 except StopMonitoringLogs: 667 events['stop_exception'].set() 668 return 669 except Exception: 670 warn(f"Error in logs callback:\n{traceback.format_exc()}") 671 672 await emit_latest_lines() 673 674 tasks = ( 675 [check_job_status_task] 676 + ([check_blocking_on_input_task] if accept_input else []) 677 + [combine_events_task] 678 ) 679 try: 680 _ = asyncio.gather(*tasks, return_exceptions=True) 681 except asyncio.exceptions.CancelledError: 682 raise 683 except Exception: 684 warn(f"Failed to run async checks:\n{traceback.format_exc()}") 685 686 watchfiles = mrsm.attempt_import('watchfiles', lazy=False) 687 dir_path_to_monitor = ( 688 _logs_path 689 or (log.file_path.parent if log else None) 690 or paths.LOGS_RESOURCES_PATH 691 ) 692 async for changes in watchfiles.awatch( 693 dir_path_to_monitor, 694 stop_event=combined_event, 695 ): 696 for change in changes: 697 file_path_str = change[1] 698 file_path = pathlib.Path(file_path_str) 699 latest_subfile_path = log.get_latest_subfile_path() 700 if latest_subfile_path != file_path: 701 continue 702 703 await emit_latest_lines() 704 705 await emit_latest_lines() 706 707 def is_blocking_on_stdin(self, debug: bool = False) -> bool: 708 """ 709 Return whether a job's daemon is blocking on stdin. 710 """ 711 if self.executor is not None: 712 return self.executor.get_job_is_blocking_on_stdin(self.name, debug=debug) 713 714 return self.is_running() and self.daemon.blocking_stdin_file_path.exists() 715 716 def get_prompt_kwargs(self, debug: bool = False) -> Dict[str, Any]: 717 """ 718 Return the kwargs to the blocking `prompt()`, if available. 719 """ 720 if self.executor is not None: 721 return self.executor.get_job_prompt_kwargs(self.name, debug=debug) 722 723 if not self.daemon.prompt_kwargs_file_path.exists(): 724 return {} 725 726 try: 727 with open(self.daemon.prompt_kwargs_file_path, 'r', encoding='utf-8') as f: 728 prompt_kwargs = json.load(f) 729 730 return prompt_kwargs 731 732 except Exception: 733 import traceback 734 traceback.print_exc() 735 return {} 736 737 def write_stdin(self, data): 738 """ 739 Write to a job's daemon's `stdin`. 740 """ 741 self.daemon.stdin_file.write(data) 742 743 @property 744 def executor(self) -> Union[Executor, None]: 745 """ 746 If the job is remote, return the connector to the remote API instance. 747 """ 748 return ( 749 mrsm.get_connector(self.executor_keys) 750 if self.executor_keys != 'local' 751 else None 752 ) 753 754 @property 755 def status(self) -> str: 756 """ 757 Return the running status of the job's daemon. 758 """ 759 if '_status_hook' in self.__dict__: 760 return self._status_hook() 761 762 if self.executor is not None: 763 return self.executor.get_job_status(self.name) 764 765 return self.daemon.status 766 767 @property 768 def pid(self) -> Union[int, None]: 769 """ 770 Return the PID of the job's dameon. 771 """ 772 if self.executor is not None: 773 return self.executor.get_job_metadata(self.name).get('daemon', {}).get('pid', None) 774 775 return self.daemon.pid 776 777 @property 778 def restart(self) -> bool: 779 """ 780 Return whether to restart a stopped job. 781 """ 782 if self.executor is not None: 783 return self.executor.get_job_metadata(self.name).get('restart', False) 784 785 return self.daemon.properties.get('restart', False) 786 787 @property 788 def result(self) -> SuccessTuple: 789 """ 790 Return the `SuccessTuple` when the job has terminated. 791 """ 792 if self.is_running(): 793 return True, f"{self} is running." 794 795 if '_result_hook' in self.__dict__: 796 return self._result_hook() 797 798 if self.executor is not None: 799 return ( 800 self.executor.get_job_metadata(self.name) 801 .get('result', (False, "No result available.")) 802 ) 803 804 _result = self.daemon.properties.get('result', None) 805 if _result is None: 806 from meerschaum.utils.daemon.Daemon import _results 807 return _results.get(self.daemon.daemon_id, (False, "No result available.")) 808 809 return tuple(_result) 810 811 @property 812 def sysargs(self) -> List[str]: 813 """ 814 Return the sysargs to use for the Daemon. 815 """ 816 if self._sysargs: 817 return self._sysargs 818 819 if self.executor is not None: 820 return self.executor.get_job_metadata(self.name).get('sysargs', []) 821 822 target_args = self.daemon.target_args 823 if target_args is None: 824 return [] 825 self._sysargs = target_args[0] if len(target_args) > 0 else [] 826 return self._sysargs 827 828 def get_daemon_properties(self) -> Dict[str, Any]: 829 """ 830 Return the `properties` dictionary for the job's daemon. 831 """ 832 remote_properties = ( 833 {} 834 if self.executor is None 835 else self.executor.get_job_properties(self.name) 836 ) 837 return { 838 **remote_properties, 839 **self._properties_patch 840 } 841 842 @property 843 def daemon(self) -> 'Daemon': 844 """ 845 Return the daemon which this job manages. 846 """ 847 from meerschaum.utils.daemon import Daemon 848 if self._daemon is not None and self.executor is None and self._sysargs: 849 return self._daemon 850 851 self._daemon = Daemon( 852 target=entry, 853 target_args=[self._sysargs], 854 target_kw={}, 855 daemon_id=self.name, 856 label=shlex.join(self._sysargs), 857 properties=self.get_daemon_properties(), 858 ) 859 if '_rotating_log' in self.__dict__: 860 self._daemon._rotating_log = self._rotating_log 861 862 if '_stdin_file' in self.__dict__: 863 self._daemon._stdin_file = self._stdin_file 864 self._daemon._blocking_stdin_file_path = self._stdin_file.blocking_file_path 865 866 return self._daemon 867 868 @property 869 def began(self) -> Union[datetime, None]: 870 """ 871 The datetime when the job began running. 872 """ 873 if self.executor is not None: 874 began_str = self.executor.get_job_began(self.name) 875 if began_str is None: 876 return None 877 return ( 878 datetime.fromisoformat(began_str) 879 .astimezone(timezone.utc) 880 .replace(tzinfo=None) 881 ) 882 883 began_str = self.daemon.properties.get('process', {}).get('began', None) 884 if began_str is None: 885 return None 886 887 return datetime.fromisoformat(began_str) 888 889 @property 890 def ended(self) -> Union[datetime, None]: 891 """ 892 The datetime when the job stopped running. 893 """ 894 if self.executor is not None: 895 ended_str = self.executor.get_job_ended(self.name) 896 if ended_str is None: 897 return None 898 return ( 899 datetime.fromisoformat(ended_str) 900 .astimezone(timezone.utc) 901 .replace(tzinfo=None) 902 ) 903 904 ended_str = self.daemon.properties.get('process', {}).get('ended', None) 905 if ended_str is None: 906 return None 907 908 return datetime.fromisoformat(ended_str) 909 910 @property 911 def paused(self) -> Union[datetime, None]: 912 """ 913 The datetime when the job was suspended while running. 914 """ 915 if self.executor is not None: 916 paused_str = self.executor.get_job_paused(self.name) 917 if paused_str is None: 918 return None 919 return ( 920 datetime.fromisoformat(paused_str) 921 .astimezone(timezone.utc) 922 .replace(tzinfo=None) 923 ) 924 925 paused_str = self.daemon.properties.get('process', {}).get('paused', None) 926 if paused_str is None: 927 return None 928 929 return datetime.fromisoformat(paused_str) 930 931 @property 932 def stop_time(self) -> Union[datetime, None]: 933 """ 934 Return the timestamp when the job was manually stopped. 935 """ 936 if self.executor is not None: 937 return self.executor.get_job_stop_time(self.name) 938 939 if not self.daemon.stop_path.exists(): 940 return None 941 942 stop_data = self.daemon._read_stop_file() 943 if not stop_data: 944 return None 945 946 stop_time_str = stop_data.get('stop_time', None) 947 if not stop_time_str: 948 warn(f"Could not read stop time for {self}.") 949 return None 950 951 return datetime.fromisoformat(stop_time_str) 952 953 @property 954 def hidden(self) -> bool: 955 """ 956 Return a bool indicating whether this job should be displayed. 957 """ 958 return ( 959 self.name.startswith('_') 960 or self.name.startswith('.') 961 or self._is_externally_managed 962 ) 963 964 def check_restart(self) -> SuccessTuple: 965 """ 966 If `restart` is `True` and the daemon is not running, 967 restart the job. 968 Do not restart if the job was manually stopped. 969 """ 970 if self.is_running(): 971 return True, f"{self} is running." 972 973 if not self.restart: 974 return True, f"{self} does not need to be restarted." 975 976 if self.stop_time is not None: 977 return True, f"{self} was manually stopped." 978 979 return self.start() 980 981 @property 982 def label(self) -> str: 983 """ 984 Return the job's Daemon label (joined sysargs). 985 """ 986 from meerschaum._internal.arguments import compress_pipeline_sysargs 987 sysargs = compress_pipeline_sysargs(self.sysargs) 988 return shlex.join(sysargs).replace(' + ', '\n+ ').replace(' : ', '\n: ').lstrip().rstrip() 989 990 @property 991 def _externally_managed_file(self) -> pathlib.Path: 992 """ 993 Return the path to the externally managed file. 994 """ 995 return self.daemon.path / '.externally-managed' 996 997 def _set_externally_managed(self): 998 """ 999 Set this job as externally managed. 1000 """ 1001 self._externally_managed = True 1002 try: 1003 self._externally_managed_file.parent.mkdir(exist_ok=True, parents=True) 1004 self._externally_managed_file.touch() 1005 except Exception as e: 1006 warn(e) 1007 1008 @property 1009 def _is_externally_managed(self) -> bool: 1010 """ 1011 Return whether this job is externally managed. 1012 """ 1013 return self.executor_keys in (None, 'local') and ( 1014 self._externally_managed or self._externally_managed_file.exists() 1015 ) 1016 1017 @property 1018 def env(self) -> Dict[str, str]: 1019 """ 1020 Return the environment variables to set for the job's process. 1021 """ 1022 if '_env' in self.__dict__: 1023 return self.__dict__['_env'] 1024 1025 _env = self.daemon.properties.get('env', {}) 1026 default_env = { 1027 'PYTHONUNBUFFERED': '1', 1028 'LINES': str(get_config('jobs', 'terminal', 'lines')), 1029 'COLUMNS': str(get_config('jobs', 'terminal', 'columns')), 1030 STATIC_CONFIG['environment']['noninteractive']: 'true', 1031 } 1032 self._env = {**default_env, **_env} 1033 return self._env 1034 1035 @property 1036 def delete_after_completion(self) -> bool: 1037 """ 1038 Return whether this job is configured to delete itself after completion. 1039 """ 1040 if '_delete_after_completion' in self.__dict__: 1041 return self.__dict__.get('_delete_after_completion', False) 1042 1043 self._delete_after_completion = self.daemon.properties.get('delete_after_completion', False) 1044 return self._delete_after_completion 1045 1046 def __str__(self) -> str: 1047 sysargs = self.sysargs 1048 sysargs_str = shlex.join(sysargs) if sysargs else '' 1049 job_str = f'Job("{self.name}"' 1050 if sysargs_str: 1051 job_str += f', "{sysargs_str}"' 1052 1053 job_str += ')' 1054 return job_str 1055 1056 def __repr__(self) -> str: 1057 return str(self) 1058 1059 def __hash__(self) -> int: 1060 return hash(self.name)
Manage a meerschaum.utils.daemon.Daemon, locally or remotely via the API.
75 def __init__( 76 self, 77 name: str, 78 sysargs: Union[List[str], str, None] = None, 79 env: Optional[Dict[str, str]] = None, 80 executor_keys: Optional[str] = None, 81 delete_after_completion: bool = False, 82 refresh_seconds: Union[int, float, None] = None, 83 _properties: Optional[Dict[str, Any]] = None, 84 _rotating_log=None, 85 _stdin_file=None, 86 _status_hook: Optional[Callable[[], str]] = None, 87 _result_hook: Optional[Callable[[], SuccessTuple]] = None, 88 _externally_managed: bool = False, 89 ): 90 """ 91 Create a new job to manage a `meerschaum.utils.daemon.Daemon`. 92 93 Parameters 94 ---------- 95 name: str 96 The name of the job to be created. 97 This will also be used as the Daemon ID. 98 99 sysargs: Union[List[str], str, None], default None 100 The sysargs of the command to be executed, e.g. 'start api'. 101 102 env: Optional[Dict[str, str]], default None 103 If provided, set these environment variables in the job's process. 104 105 executor_keys: Optional[str], default None 106 If provided, execute the job remotely on an API instance, e.g. 'api:main'. 107 108 delete_after_completion: bool, default False 109 If `True`, delete this job when it has finished executing. 110 111 refresh_seconds: Union[int, float, None], default None 112 The number of seconds to sleep between refreshes. 113 Defaults to the configured value `system.cli.refresh_seconds`. 114 115 _properties: Optional[Dict[str, Any]], default None 116 If provided, use this to patch the daemon's properties. 117 """ 118 from meerschaum.utils.daemon import Daemon 119 for char in BANNED_CHARS: 120 if char in name: 121 raise ValueError(f"Invalid name: ({char}) is not allowed.") 122 123 if isinstance(sysargs, str): 124 sysargs = shlex.split(sysargs) 125 126 and_key = STATIC_CONFIG['system']['arguments']['and_key'] 127 escaped_and_key = STATIC_CONFIG['system']['arguments']['escaped_and_key'] 128 if sysargs: 129 sysargs = [ 130 (arg if arg != escaped_and_key else and_key) 131 for arg in sysargs 132 ] 133 134 ### NOTE: 'local' and 'systemd' executors are being coalesced. 135 if executor_keys is None: 136 from meerschaum.jobs import get_executor_keys_from_context 137 executor_keys = get_executor_keys_from_context() 138 139 self.executor_keys = executor_keys 140 self.name = name 141 self.refresh_seconds = ( 142 refresh_seconds 143 if refresh_seconds is not None 144 else mrsm.get_config('system', 'cli', 'refresh_seconds') 145 ) 146 try: 147 self._daemon = ( 148 Daemon(daemon_id=name) 149 if executor_keys == 'local' 150 else None 151 ) 152 except Exception: 153 self._daemon = None 154 155 ### Handle any injected dependencies. 156 if _rotating_log is not None: 157 self._rotating_log = _rotating_log 158 if self._daemon is not None: 159 self._daemon._rotating_log = _rotating_log 160 161 if _stdin_file is not None: 162 self._stdin_file = _stdin_file 163 if self._daemon is not None: 164 self._daemon._stdin_file = _stdin_file 165 self._daemon._blocking_stdin_file_path = _stdin_file.blocking_file_path 166 167 if _status_hook is not None: 168 self._status_hook = _status_hook 169 170 if _result_hook is not None: 171 self._result_hook = _result_hook 172 173 self._externally_managed = _externally_managed 174 self._properties_patch = _properties or {} 175 if _externally_managed: 176 self._properties_patch.update({'externally_managed': _externally_managed}) 177 178 if env: 179 self._properties_patch.update({'env': env}) 180 181 if delete_after_completion: 182 self._properties_patch.update({'delete_after_completion': delete_after_completion}) 183 184 daemon_sysargs = ( 185 self._daemon.properties.get('target', {}).get('args', [None])[0] 186 if self._daemon is not None 187 else None 188 ) 189 190 if daemon_sysargs and sysargs and daemon_sysargs != sysargs: 191 warn("Given sysargs differ from existing sysargs.") 192 193 self._sysargs = [ 194 arg 195 for arg in (daemon_sysargs or sysargs or []) 196 if arg not in ('-d', '--daemon') 197 ] 198 for restart_flag in RESTART_FLAGS: 199 if restart_flag in self._sysargs: 200 self._properties_patch.update({'restart': True}) 201 break
Create a new job to manage a meerschaum.utils.daemon.Daemon.
Parameters
- name (str): The name of the job to be created. This will also be used as the Daemon ID.
- sysargs (Union[List[str], str, None], default None): The sysargs of the command to be executed, e.g. 'start api'.
- env (Optional[Dict[str, str]], default None): If provided, set these environment variables in the job's process.
- executor_keys (Optional[str], default None): If provided, execute the job remotely on an API instance, e.g. 'api:main'.
- delete_after_completion (bool, default False):
If
True, delete this job when it has finished executing. - refresh_seconds (Union[int, float, None], default None):
The number of seconds to sleep between refreshes.
Defaults to the configured value
system.cli.refresh_seconds. - _properties (Optional[Dict[str, Any]], default None): If provided, use this to patch the daemon's properties.
203 @staticmethod 204 def from_pid(pid: int, executor_keys: Optional[str] = None) -> Job: 205 """ 206 Build a `Job` from the PID of a running Meerschaum process. 207 208 Parameters 209 ---------- 210 pid: int 211 The PID of the process. 212 213 executor_keys: Optional[str], default None 214 The executor keys to assign to the job. 215 """ 216 psutil = mrsm.attempt_import('psutil') 217 try: 218 process = psutil.Process(pid) 219 except psutil.NoSuchProcess as e: 220 warn(f"Process with PID {pid} does not exist.", stack=False) 221 raise e 222 223 command_args = process.cmdline() 224 is_daemon = command_args[1] == '-c' 225 226 if is_daemon: 227 daemon_id = command_args[-1].split('daemon_id=')[-1].split(')')[0].replace("'", '') 228 root_dir = process.environ().get(STATIC_CONFIG['environment']['root'], None) 229 if root_dir is None: 230 root_dir = paths.ROOT_DIR_PATH 231 else: 232 root_dir = pathlib.Path(root_dir) 233 jobs_dir = root_dir / paths.DAEMON_RESOURCES_PATH.name 234 daemon_dir = jobs_dir / daemon_id 235 pid_file = daemon_dir / 'process.pid' 236 237 if pid_file.exists(): 238 with open(pid_file, 'r', encoding='utf-8') as f: 239 daemon_pid = int(f.read()) 240 241 if pid != daemon_pid: 242 raise EnvironmentError(f"Differing PIDs: {pid=}, {daemon_pid=}") 243 else: 244 raise EnvironmentError(f"Is job '{daemon_id}' running?") 245 246 return Job(daemon_id, executor_keys=executor_keys) 247 248 from meerschaum._internal.arguments._parse_arguments import parse_arguments 249 from meerschaum.utils.daemon import get_new_daemon_name 250 251 mrsm_ix = 0 252 for i, arg in enumerate(command_args): 253 if 'mrsm' in arg or 'meerschaum' in arg.lower(): 254 mrsm_ix = i 255 break 256 257 sysargs = command_args[mrsm_ix+1:] 258 kwargs = parse_arguments(sysargs) 259 name = kwargs.get('name', get_new_daemon_name()) 260 return Job(name, sysargs, executor_keys=executor_keys)
Build a Job from the PID of a running Meerschaum process.
Parameters
- pid (int): The PID of the process.
- executor_keys (Optional[str], default None): The executor keys to assign to the job.
262 def start(self, debug: bool = False) -> SuccessTuple: 263 """ 264 Start the job's daemon. 265 """ 266 if self.executor is not None: 267 if not self.exists(debug=debug): 268 return self.executor.create_job( 269 self.name, 270 self.sysargs, 271 properties=self.daemon.properties, 272 debug=debug, 273 ) 274 return self.executor.start_job(self.name, debug=debug) 275 276 if self.is_running(): 277 return True, f"{self} is already running." 278 279 success, msg = self.daemon.run( 280 keep_daemon_output=(not self.delete_after_completion), 281 allow_dirty_run=True, 282 ) 283 if not success: 284 return success, msg 285 286 return success, f"Started {self}."
Start the job's daemon.
288 def stop( 289 self, 290 timeout_seconds: Union[int, float, None] = None, 291 debug: bool = False, 292 ) -> SuccessTuple: 293 """ 294 Stop the job's daemon. 295 """ 296 if self.executor is not None: 297 return self.executor.stop_job(self.name, debug=debug) 298 299 if self.daemon.status == 'stopped': 300 if not self.restart: 301 return True, f"{self} is not running." 302 elif self.stop_time is not None: 303 return True, f"{self} will not restart until manually started." 304 305 quit_success, quit_msg = self.daemon.quit(timeout=timeout_seconds) 306 if quit_success: 307 return quit_success, f"Stopped {self}." 308 309 warn( 310 f"Failed to gracefully quit {self}: {quit_msg}", 311 stack=False, 312 ) 313 kill_success, kill_msg = self.daemon.kill(timeout=timeout_seconds) 314 if not kill_success: 315 return kill_success, kill_msg 316 317 return kill_success, f"Killed {self}."
Stop the job's daemon.
319 def pause( 320 self, 321 timeout_seconds: Union[int, float, None] = None, 322 debug: bool = False, 323 ) -> SuccessTuple: 324 """ 325 Pause the job's daemon. 326 """ 327 if self.executor is not None: 328 return self.executor.pause_job(self.name, debug=debug) 329 330 pause_success, pause_msg = self.daemon.pause(timeout=timeout_seconds) 331 if not pause_success: 332 return pause_success, pause_msg 333 334 return pause_success, f"Paused {self}."
Pause the job's daemon.
336 def delete(self, debug: bool = False) -> SuccessTuple: 337 """ 338 Delete the job and its daemon. 339 """ 340 if self.executor is not None: 341 return self.executor.delete_job(self.name, debug=debug) 342 343 if self.is_running(): 344 stop_success, stop_msg = self.stop() 345 if not stop_success: 346 return stop_success, stop_msg 347 348 cleanup_success, cleanup_msg = self.daemon.cleanup() 349 if not cleanup_success: 350 return cleanup_success, cleanup_msg 351 352 _ = self.daemon._properties.pop('result', None) 353 return cleanup_success, f"Deleted {self}."
Delete the job and its daemon.
355 def is_running(self) -> bool: 356 """ 357 Determine whether the job's daemon is running. 358 """ 359 return self.status == 'running'
Determine whether the job's daemon is running.
361 def exists(self, debug: bool = False) -> bool: 362 """ 363 Determine whether the job exists. 364 """ 365 if self.executor is not None: 366 return self.executor.get_job_exists(self.name, debug=debug) 367 368 return self.daemon.path.exists()
Determine whether the job exists.
370 def get_logs(self) -> Union[str, None]: 371 """ 372 Return the output text of the job's daemon. 373 """ 374 if self.executor is not None: 375 return self.executor.get_logs(self.name) 376 377 return self.daemon.log_text
Return the output text of the job's daemon.
379 def monitor_logs( 380 self, 381 callback_function: Callable[[str], None] = _default_stdout_callback, 382 input_callback_function: Optional[Callable[[], str]] = None, 383 stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None, 384 stop_event: Optional[asyncio.Event] = None, 385 stop_on_exit: bool = False, 386 strip_timestamps: bool = False, 387 accept_input: bool = True, 388 debug: bool = False, 389 _logs_path: Optional[pathlib.Path] = None, 390 _log=None, 391 _stdin_file=None, 392 _wait_if_stopped: bool = True, 393 ): 394 """ 395 Monitor the job's log files and execute a callback on new lines. 396 397 Parameters 398 ---------- 399 callback_function: Callable[[str], None], default partial(print, end='') 400 The callback to execute as new data comes in. 401 Defaults to printing the output directly to `stdout`. 402 403 input_callback_function: Optional[Callable[[], str]], default None 404 If provided, execute this callback when the daemon is blocking on stdin. 405 Defaults to `sys.stdin.readline()`. 406 407 stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None 408 If provided, execute this callback when the daemon stops. 409 The job's SuccessTuple will be passed to the callback. 410 411 stop_event: Optional[asyncio.Event], default None 412 If provided, stop monitoring when this event is set. 413 You may instead raise `meerschaum.jobs.StopMonitoringLogs` 414 from within `callback_function` to stop monitoring. 415 416 stop_on_exit: bool, default False 417 If `True`, stop monitoring when the job stops. 418 419 strip_timestamps: bool, default False 420 If `True`, remove leading timestamps from lines. 421 422 accept_input: bool, default True 423 If `True`, accept input when the daemon blocks on stdin. 424 """ 425 if self.executor is not None: 426 self.executor.monitor_logs( 427 self.name, 428 callback_function, 429 input_callback_function=input_callback_function, 430 stop_callback_function=stop_callback_function, 431 stop_on_exit=stop_on_exit, 432 accept_input=accept_input, 433 strip_timestamps=strip_timestamps, 434 debug=debug, 435 ) 436 return 437 438 monitor_logs_coroutine = self.monitor_logs_async( 439 callback_function=callback_function, 440 input_callback_function=input_callback_function, 441 stop_callback_function=stop_callback_function, 442 stop_event=stop_event, 443 stop_on_exit=stop_on_exit, 444 strip_timestamps=strip_timestamps, 445 accept_input=accept_input, 446 debug=debug, 447 _logs_path=_logs_path, 448 _log=_log, 449 _stdin_file=_stdin_file, 450 _wait_if_stopped=_wait_if_stopped, 451 ) 452 return asyncio.run(monitor_logs_coroutine)
Monitor the job's log files and execute a callback on new lines.
Parameters
- callback_function (Callable[[str], None], default partial(print, end='')):
The callback to execute as new data comes in.
Defaults to printing the output directly to
stdout. - input_callback_function (Optional[Callable[[], str]], default None):
If provided, execute this callback when the daemon is blocking on stdin.
Defaults to
sys.stdin.readline(). - stop_callback_function (Optional[Callable[[SuccessTuple]], str], default None): If provided, execute this callback when the daemon stops. The job's SuccessTuple will be passed to the callback.
- stop_event (Optional[asyncio.Event], default None):
If provided, stop monitoring when this event is set.
You may instead raise
meerschaum.jobs.StopMonitoringLogsfrom withincallback_functionto stop monitoring. - stop_on_exit (bool, default False):
If
True, stop monitoring when the job stops. - strip_timestamps (bool, default False):
If
True, remove leading timestamps from lines. - accept_input (bool, default True):
If
True, accept input when the daemon blocks on stdin.
454 async def monitor_logs_async( 455 self, 456 callback_function: Callable[[str], None] = _default_stdout_callback, 457 input_callback_function: Optional[Callable[[], str]] = None, 458 stop_callback_function: Optional[Callable[[SuccessTuple], None]] = None, 459 stop_event: Optional[asyncio.Event] = None, 460 stop_on_exit: bool = False, 461 strip_timestamps: bool = False, 462 accept_input: bool = True, 463 debug: bool = False, 464 _logs_path: Optional[pathlib.Path] = None, 465 _log=None, 466 _stdin_file=None, 467 _wait_if_stopped: bool = True, 468 ): 469 """ 470 Monitor the job's log files and await a callback on new lines. 471 472 Parameters 473 ---------- 474 callback_function: Callable[[str], None], default _default_stdout_callback 475 The callback to execute as new data comes in. 476 Defaults to printing the output directly to `stdout`. 477 478 input_callback_function: Optional[Callable[[], str]], default None 479 If provided, execute this callback when the daemon is blocking on stdin. 480 Defaults to `sys.stdin.readline()`. 481 482 stop_callback_function: Optional[Callable[[SuccessTuple]], str], default None 483 If provided, execute this callback when the daemon stops. 484 The job's SuccessTuple will be passed to the callback. 485 486 stop_event: Optional[asyncio.Event], default None 487 If provided, stop monitoring when this event is set. 488 You may instead raise `meerschaum.jobs.StopMonitoringLogs` 489 from within `callback_function` to stop monitoring. 490 491 stop_on_exit: bool, default False 492 If `True`, stop monitoring when the job stops. 493 494 strip_timestamps: bool, default False 495 If `True`, remove leading timestamps from lines. 496 497 accept_input: bool, default True 498 If `True`, accept input when the daemon blocks on stdin. 499 """ 500 from meerschaum.utils.prompt import prompt 501 502 def default_input_callback_function(): 503 prompt_kwargs = self.get_prompt_kwargs(debug=debug) 504 if prompt_kwargs: 505 answer = prompt(**prompt_kwargs) 506 return answer + '\n' 507 return sys.stdin.readline() 508 509 if input_callback_function is None: 510 input_callback_function = default_input_callback_function 511 512 if self.executor is not None: 513 await self.executor.monitor_logs_async( 514 self.name, 515 callback_function, 516 input_callback_function=input_callback_function, 517 stop_callback_function=stop_callback_function, 518 stop_on_exit=stop_on_exit, 519 strip_timestamps=strip_timestamps, 520 accept_input=accept_input, 521 debug=debug, 522 ) 523 return 524 525 from meerschaum.utils.formatting._jobs import strip_timestamp_from_line 526 527 events = { 528 'user': stop_event, 529 'stopped': asyncio.Event(), 530 'stop_token': asyncio.Event(), 531 'stop_exception': asyncio.Event(), 532 'stopped_timeout': asyncio.Event(), 533 } 534 combined_event = asyncio.Event() 535 emitted_text = False 536 stdin_file = _stdin_file if _stdin_file is not None else self.daemon.stdin_file 537 538 async def check_job_status(): 539 if not stop_on_exit: 540 return 541 542 nonlocal emitted_text 543 544 sleep_time = 0.1 545 while sleep_time < 0.2: 546 if self.status == 'stopped': 547 if not emitted_text and _wait_if_stopped: 548 await asyncio.sleep(sleep_time) 549 sleep_time = round(sleep_time * 1.1, 3) 550 continue 551 552 if stop_callback_function is not None: 553 try: 554 if asyncio.iscoroutinefunction(stop_callback_function): 555 await stop_callback_function(self.result) 556 else: 557 stop_callback_function(self.result) 558 except asyncio.exceptions.CancelledError: 559 break 560 except Exception: 561 warn(traceback.format_exc()) 562 563 if stop_on_exit: 564 events['stopped'].set() 565 566 break 567 await asyncio.sleep(0.1) 568 569 events['stopped_timeout'].set() 570 571 async def check_blocking_on_input(): 572 while True: 573 if not emitted_text or not self.is_blocking_on_stdin(): 574 try: 575 await asyncio.sleep(self.refresh_seconds) 576 except asyncio.exceptions.CancelledError: 577 break 578 continue 579 580 if not self.is_running(): 581 break 582 583 await emit_latest_lines() 584 585 try: 586 print('', end='', flush=True) 587 if asyncio.iscoroutinefunction(input_callback_function): 588 data = await input_callback_function() 589 else: 590 loop = asyncio.get_running_loop() 591 data = await loop.run_in_executor(None, input_callback_function) 592 except KeyboardInterrupt: 593 break 594 # if not data.endswith('\n'): 595 # data += '\n' 596 597 stdin_file.write(data) 598 await asyncio.sleep(self.refresh_seconds) 599 600 async def combine_events(): 601 event_tasks = [ 602 asyncio.create_task(event.wait()) 603 for event in events.values() 604 if event is not None 605 ] 606 if not event_tasks: 607 return 608 609 try: 610 done, pending = await asyncio.wait( 611 event_tasks, 612 return_when=asyncio.FIRST_COMPLETED, 613 ) 614 for task in pending: 615 task.cancel() 616 except asyncio.exceptions.CancelledError: 617 pass 618 finally: 619 combined_event.set() 620 621 check_job_status_task = asyncio.create_task(check_job_status()) 622 check_blocking_on_input_task = asyncio.create_task(check_blocking_on_input()) 623 combine_events_task = asyncio.create_task(combine_events()) 624 625 log = _log if _log is not None else self.daemon.rotating_log 626 lines_to_show = ( 627 self.daemon.properties.get( 628 'logs', {} 629 ).get( 630 'lines_to_show', get_config('jobs', 'logs', 'lines_to_show') 631 ) 632 ) 633 634 async def emit_latest_lines(): 635 nonlocal emitted_text 636 nonlocal stop_event 637 lines = log.readlines() 638 for line in lines[(-1 * lines_to_show):]: 639 if stop_event is not None and stop_event.is_set(): 640 return 641 642 line_stripped_extra = strip_timestamp_from_line(line.strip()) 643 line_stripped = strip_timestamp_from_line(line) 644 645 if line_stripped_extra == STOP_TOKEN: 646 events['stop_token'].set() 647 return 648 649 if line_stripped_extra == CLEAR_TOKEN: 650 clear_screen(debug=debug) 651 continue 652 653 if line_stripped_extra == FLUSH_TOKEN.strip(): 654 line_stripped = '' 655 line = '' 656 657 if strip_timestamps: 658 line = line_stripped 659 660 try: 661 if asyncio.iscoroutinefunction(callback_function): 662 await callback_function(line) 663 else: 664 callback_function(line) 665 emitted_text = True 666 except StopMonitoringLogs: 667 events['stop_exception'].set() 668 return 669 except Exception: 670 warn(f"Error in logs callback:\n{traceback.format_exc()}") 671 672 await emit_latest_lines() 673 674 tasks = ( 675 [check_job_status_task] 676 + ([check_blocking_on_input_task] if accept_input else []) 677 + [combine_events_task] 678 ) 679 try: 680 _ = asyncio.gather(*tasks, return_exceptions=True) 681 except asyncio.exceptions.CancelledError: 682 raise 683 except Exception: 684 warn(f"Failed to run async checks:\n{traceback.format_exc()}") 685 686 watchfiles = mrsm.attempt_import('watchfiles', lazy=False) 687 dir_path_to_monitor = ( 688 _logs_path 689 or (log.file_path.parent if log else None) 690 or paths.LOGS_RESOURCES_PATH 691 ) 692 async for changes in watchfiles.awatch( 693 dir_path_to_monitor, 694 stop_event=combined_event, 695 ): 696 for change in changes: 697 file_path_str = change[1] 698 file_path = pathlib.Path(file_path_str) 699 latest_subfile_path = log.get_latest_subfile_path() 700 if latest_subfile_path != file_path: 701 continue 702 703 await emit_latest_lines() 704 705 await emit_latest_lines()
Monitor the job's log files and await a callback on new lines.
Parameters
- callback_function (Callable[[str], None], default _default_stdout_callback):
The callback to execute as new data comes in.
Defaults to printing the output directly to
stdout. - input_callback_function (Optional[Callable[[], str]], default None):
If provided, execute this callback when the daemon is blocking on stdin.
Defaults to
sys.stdin.readline(). - stop_callback_function (Optional[Callable[[SuccessTuple]], str], default None): If provided, execute this callback when the daemon stops. The job's SuccessTuple will be passed to the callback.
- stop_event (Optional[asyncio.Event], default None):
If provided, stop monitoring when this event is set.
You may instead raise
meerschaum.jobs.StopMonitoringLogsfrom withincallback_functionto stop monitoring. - stop_on_exit (bool, default False):
If
True, stop monitoring when the job stops. - strip_timestamps (bool, default False):
If
True, remove leading timestamps from lines. - accept_input (bool, default True):
If
True, accept input when the daemon blocks on stdin.
707 def is_blocking_on_stdin(self, debug: bool = False) -> bool: 708 """ 709 Return whether a job's daemon is blocking on stdin. 710 """ 711 if self.executor is not None: 712 return self.executor.get_job_is_blocking_on_stdin(self.name, debug=debug) 713 714 return self.is_running() and self.daemon.blocking_stdin_file_path.exists()
Return whether a job's daemon is blocking on stdin.
716 def get_prompt_kwargs(self, debug: bool = False) -> Dict[str, Any]: 717 """ 718 Return the kwargs to the blocking `prompt()`, if available. 719 """ 720 if self.executor is not None: 721 return self.executor.get_job_prompt_kwargs(self.name, debug=debug) 722 723 if not self.daemon.prompt_kwargs_file_path.exists(): 724 return {} 725 726 try: 727 with open(self.daemon.prompt_kwargs_file_path, 'r', encoding='utf-8') as f: 728 prompt_kwargs = json.load(f) 729 730 return prompt_kwargs 731 732 except Exception: 733 import traceback 734 traceback.print_exc() 735 return {}
Return the kwargs to the blocking prompt(), if available.
737 def write_stdin(self, data): 738 """ 739 Write to a job's daemon's `stdin`. 740 """ 741 self.daemon.stdin_file.write(data)
Write to a job's daemon's stdin.
743 @property 744 def executor(self) -> Union[Executor, None]: 745 """ 746 If the job is remote, return the connector to the remote API instance. 747 """ 748 return ( 749 mrsm.get_connector(self.executor_keys) 750 if self.executor_keys != 'local' 751 else None 752 )
If the job is remote, return the connector to the remote API instance.
754 @property 755 def status(self) -> str: 756 """ 757 Return the running status of the job's daemon. 758 """ 759 if '_status_hook' in self.__dict__: 760 return self._status_hook() 761 762 if self.executor is not None: 763 return self.executor.get_job_status(self.name) 764 765 return self.daemon.status
Return the running status of the job's daemon.
767 @property 768 def pid(self) -> Union[int, None]: 769 """ 770 Return the PID of the job's dameon. 771 """ 772 if self.executor is not None: 773 return self.executor.get_job_metadata(self.name).get('daemon', {}).get('pid', None) 774 775 return self.daemon.pid
Return the PID of the job's dameon.
777 @property 778 def restart(self) -> bool: 779 """ 780 Return whether to restart a stopped job. 781 """ 782 if self.executor is not None: 783 return self.executor.get_job_metadata(self.name).get('restart', False) 784 785 return self.daemon.properties.get('restart', False)
Return whether to restart a stopped job.
787 @property 788 def result(self) -> SuccessTuple: 789 """ 790 Return the `SuccessTuple` when the job has terminated. 791 """ 792 if self.is_running(): 793 return True, f"{self} is running." 794 795 if '_result_hook' in self.__dict__: 796 return self._result_hook() 797 798 if self.executor is not None: 799 return ( 800 self.executor.get_job_metadata(self.name) 801 .get('result', (False, "No result available.")) 802 ) 803 804 _result = self.daemon.properties.get('result', None) 805 if _result is None: 806 from meerschaum.utils.daemon.Daemon import _results 807 return _results.get(self.daemon.daemon_id, (False, "No result available.")) 808 809 return tuple(_result)
Return the SuccessTuple when the job has terminated.
811 @property 812 def sysargs(self) -> List[str]: 813 """ 814 Return the sysargs to use for the Daemon. 815 """ 816 if self._sysargs: 817 return self._sysargs 818 819 if self.executor is not None: 820 return self.executor.get_job_metadata(self.name).get('sysargs', []) 821 822 target_args = self.daemon.target_args 823 if target_args is None: 824 return [] 825 self._sysargs = target_args[0] if len(target_args) > 0 else [] 826 return self._sysargs
Return the sysargs to use for the Daemon.
828 def get_daemon_properties(self) -> Dict[str, Any]: 829 """ 830 Return the `properties` dictionary for the job's daemon. 831 """ 832 remote_properties = ( 833 {} 834 if self.executor is None 835 else self.executor.get_job_properties(self.name) 836 ) 837 return { 838 **remote_properties, 839 **self._properties_patch 840 }
Return the properties dictionary for the job's daemon.
842 @property 843 def daemon(self) -> 'Daemon': 844 """ 845 Return the daemon which this job manages. 846 """ 847 from meerschaum.utils.daemon import Daemon 848 if self._daemon is not None and self.executor is None and self._sysargs: 849 return self._daemon 850 851 self._daemon = Daemon( 852 target=entry, 853 target_args=[self._sysargs], 854 target_kw={}, 855 daemon_id=self.name, 856 label=shlex.join(self._sysargs), 857 properties=self.get_daemon_properties(), 858 ) 859 if '_rotating_log' in self.__dict__: 860 self._daemon._rotating_log = self._rotating_log 861 862 if '_stdin_file' in self.__dict__: 863 self._daemon._stdin_file = self._stdin_file 864 self._daemon._blocking_stdin_file_path = self._stdin_file.blocking_file_path 865 866 return self._daemon
Return the daemon which this job manages.
868 @property 869 def began(self) -> Union[datetime, None]: 870 """ 871 The datetime when the job began running. 872 """ 873 if self.executor is not None: 874 began_str = self.executor.get_job_began(self.name) 875 if began_str is None: 876 return None 877 return ( 878 datetime.fromisoformat(began_str) 879 .astimezone(timezone.utc) 880 .replace(tzinfo=None) 881 ) 882 883 began_str = self.daemon.properties.get('process', {}).get('began', None) 884 if began_str is None: 885 return None 886 887 return datetime.fromisoformat(began_str)
The datetime when the job began running.
889 @property 890 def ended(self) -> Union[datetime, None]: 891 """ 892 The datetime when the job stopped running. 893 """ 894 if self.executor is not None: 895 ended_str = self.executor.get_job_ended(self.name) 896 if ended_str is None: 897 return None 898 return ( 899 datetime.fromisoformat(ended_str) 900 .astimezone(timezone.utc) 901 .replace(tzinfo=None) 902 ) 903 904 ended_str = self.daemon.properties.get('process', {}).get('ended', None) 905 if ended_str is None: 906 return None 907 908 return datetime.fromisoformat(ended_str)
The datetime when the job stopped running.
910 @property 911 def paused(self) -> Union[datetime, None]: 912 """ 913 The datetime when the job was suspended while running. 914 """ 915 if self.executor is not None: 916 paused_str = self.executor.get_job_paused(self.name) 917 if paused_str is None: 918 return None 919 return ( 920 datetime.fromisoformat(paused_str) 921 .astimezone(timezone.utc) 922 .replace(tzinfo=None) 923 ) 924 925 paused_str = self.daemon.properties.get('process', {}).get('paused', None) 926 if paused_str is None: 927 return None 928 929 return datetime.fromisoformat(paused_str)
The datetime when the job was suspended while running.
931 @property 932 def stop_time(self) -> Union[datetime, None]: 933 """ 934 Return the timestamp when the job was manually stopped. 935 """ 936 if self.executor is not None: 937 return self.executor.get_job_stop_time(self.name) 938 939 if not self.daemon.stop_path.exists(): 940 return None 941 942 stop_data = self.daemon._read_stop_file() 943 if not stop_data: 944 return None 945 946 stop_time_str = stop_data.get('stop_time', None) 947 if not stop_time_str: 948 warn(f"Could not read stop time for {self}.") 949 return None 950 951 return datetime.fromisoformat(stop_time_str)
Return the timestamp when the job was manually stopped.
964 def check_restart(self) -> SuccessTuple: 965 """ 966 If `restart` is `True` and the daemon is not running, 967 restart the job. 968 Do not restart if the job was manually stopped. 969 """ 970 if self.is_running(): 971 return True, f"{self} is running." 972 973 if not self.restart: 974 return True, f"{self} does not need to be restarted." 975 976 if self.stop_time is not None: 977 return True, f"{self} was manually stopped." 978 979 return self.start()
If restart is True and the daemon is not running,
restart the job.
Do not restart if the job was manually stopped.
981 @property 982 def label(self) -> str: 983 """ 984 Return the job's Daemon label (joined sysargs). 985 """ 986 from meerschaum._internal.arguments import compress_pipeline_sysargs 987 sysargs = compress_pipeline_sysargs(self.sysargs) 988 return shlex.join(sysargs).replace(' + ', '\n+ ').replace(' : ', '\n: ').lstrip().rstrip()
Return the job's Daemon label (joined sysargs).
1017 @property 1018 def env(self) -> Dict[str, str]: 1019 """ 1020 Return the environment variables to set for the job's process. 1021 """ 1022 if '_env' in self.__dict__: 1023 return self.__dict__['_env'] 1024 1025 _env = self.daemon.properties.get('env', {}) 1026 default_env = { 1027 'PYTHONUNBUFFERED': '1', 1028 'LINES': str(get_config('jobs', 'terminal', 'lines')), 1029 'COLUMNS': str(get_config('jobs', 'terminal', 'columns')), 1030 STATIC_CONFIG['environment']['noninteractive']: 'true', 1031 } 1032 self._env = {**default_env, **_env} 1033 return self._env
Return the environment variables to set for the job's process.
1035 @property 1036 def delete_after_completion(self) -> bool: 1037 """ 1038 Return whether this job is configured to delete itself after completion. 1039 """ 1040 if '_delete_after_completion' in self.__dict__: 1041 return self.__dict__.get('_delete_after_completion', False) 1042 1043 self._delete_after_completion = self.daemon.properties.get('delete_after_completion', False) 1044 return self._delete_after_completion
Return whether this job is configured to delete itself after completion.
10def pprint( 11 *args, 12 detect_password: bool = True, 13 nopretty: bool = False, 14 **kw 15) -> None: 16 """Pretty print an object according to the configured ANSI and UNICODE settings. 17 If detect_password is True (default), search and replace passwords with '*' characters. 18 Does not mutate objects. 19 """ 20 import copy 21 import json 22 from meerschaum.utils.packages import attempt_import, import_rich 23 from meerschaum.utils.formatting import ANSI, get_console, print_tuple 24 from meerschaum.utils.warnings import error 25 from meerschaum.utils.misc import replace_password, dict_from_od, filter_keywords 26 from collections import OrderedDict 27 28 if ( 29 len(args) == 1 30 and 31 isinstance(args[0], tuple) 32 and 33 len(args[0]) == 2 34 and 35 isinstance(args[0][0], bool) 36 and 37 isinstance(args[0][1], str) 38 ): 39 return print_tuple(args[0], **filter_keywords(print_tuple, **kw)) 40 41 modify = True 42 rich_pprint = None 43 if ANSI and not nopretty: 44 rich = import_rich() 45 if rich is not None: 46 rich_pretty = attempt_import('rich.pretty') 47 if rich_pretty is not None: 48 def _rich_pprint(*args, **kw): 49 _console = get_console() 50 _kw = filter_keywords(_console.print, **kw) 51 _console.print(*args, **_kw) 52 rich_pprint = _rich_pprint 53 elif not nopretty: 54 pprintpp = attempt_import('pprintpp', warn=False) 55 try: 56 _pprint = pprintpp.pprint 57 except Exception : 58 import pprint as _pprint_module 59 _pprint = _pprint_module.pprint 60 61 func = ( 62 _pprint if rich_pprint is None else rich_pprint 63 ) if not nopretty else print 64 65 try: 66 args_copy = copy.deepcopy(args) 67 except Exception: 68 args_copy = args 69 modify = False 70 71 _args = [] 72 for a in args: 73 c = a 74 ### convert OrderedDict into dict 75 if isinstance(a, OrderedDict) or issubclass(type(a), OrderedDict): 76 c = dict_from_od(copy.deepcopy(c)) 77 _args.append(c) 78 args = _args 79 80 _args = list(args) 81 if detect_password and modify: 82 _args = [] 83 for a in args: 84 c = a 85 if isinstance(c, dict): 86 c = replace_password(copy.deepcopy(c)) 87 if nopretty: 88 try: 89 c = json.dumps(c) 90 is_json = True 91 except Exception: 92 is_json = False 93 if not is_json: 94 try: 95 c = str(c) 96 except Exception: 97 pass 98 _args.append(c) 99 100 ### filter out unsupported keywords 101 func_kw = filter_keywords(func, **kw) if not nopretty else {} 102 error_msg = None 103 try: 104 func(*_args, **func_kw) 105 except Exception as e: 106 error_msg = e 107 if error_msg is not None: 108 error(error_msg)
Pretty print an object according to the configured ANSI and UNICODE settings. If detect_password is True (default), search and replace passwords with '*' characters. Does not mutate objects.
1418def attempt_import( 1419 *names: str, 1420 lazy: bool = True, 1421 warn: bool = True, 1422 install: bool = True, 1423 venv: Optional[str] = 'mrsm', 1424 precheck: bool = True, 1425 split: bool = True, 1426 check_update: bool = False, 1427 check_pypi: bool = False, 1428 check_is_installed: bool = True, 1429 allow_outside_venv: bool = True, 1430 color: bool = True, 1431 debug: bool = False 1432) -> Any: 1433 """ 1434 Raise a warning if packages are not installed; otherwise import and return modules. 1435 If `lazy` is `True`, return lazy-imported modules. 1436 1437 Returns tuple of modules if multiple names are provided, else returns one module. 1438 1439 Parameters 1440 ---------- 1441 names: List[str] 1442 The packages to be imported. 1443 1444 lazy: bool, default True 1445 If `True`, lazily load packages. 1446 1447 warn: bool, default True 1448 If `True`, raise a warning if a package cannot be imported. 1449 1450 install: bool, default True 1451 If `True`, attempt to install a missing package into the designated virtual environment. 1452 If `check_update` is True, install updates if available. 1453 1454 venv: Optional[str], default 'mrsm' 1455 The virtual environment in which to search for packages and to install packages into. 1456 1457 precheck: bool, default True 1458 If `True`, attempt to find module before importing (necessary for checking if modules exist 1459 and retaining lazy imports), otherwise assume lazy is `False`. 1460 1461 split: bool, default True 1462 If `True`, split packages' names on `'.'`. 1463 1464 check_update: bool, default False 1465 If `True` and `install` is `True`, install updates if the required minimum version 1466 does not match. 1467 1468 check_pypi: bool, default False 1469 If `True` and `check_update` is `True`, check PyPI when determining whether 1470 an update is required. 1471 1472 check_is_installed: bool, default True 1473 If `True`, check if the package is contained in the virtual environment. 1474 1475 allow_outside_venv: bool, default True 1476 If `True`, search outside of the specified virtual environment 1477 if the package cannot be found. 1478 Setting to `False` will reinstall the package into a virtual environment, even if it 1479 is installed outside. 1480 1481 color: bool, default True 1482 If `False`, do not print ANSI colors. 1483 1484 Returns 1485 ------- 1486 The specified modules. If they're not available and `install` is `True`, it will first 1487 download them into a virtual environment and return the modules. 1488 1489 Examples 1490 -------- 1491 >>> pandas, sqlalchemy = attempt_import('pandas', 'sqlalchemy') 1492 >>> pandas = attempt_import('pandas') 1493 1494 """ 1495 1496 import importlib.util 1497 1498 global emitted_auto_install_warning 1499 no_auto_install_env_var = 'MRSM_NO_AUTO_INSTALL' 1500 if os.environ.get(no_auto_install_env_var, '').lower() in ('1', 'true', 'yes'): 1501 install = False 1502 1503 ### to prevent recursion, check if parent Meerschaum package is being imported 1504 if names == ('meerschaum',): 1505 return _import_module('meerschaum') 1506 1507 if venv == 'mrsm' and _import_hook_venv is not None: 1508 if debug: 1509 print(f"Import hook for virtual environment '{_import_hook_venv}' is active.") 1510 venv = _import_hook_venv 1511 1512 _warnings = _import_module('meerschaum.utils.warnings') 1513 warn_function = _warnings.warn 1514 1515 def do_import(_name: str, **kw) -> Union['ModuleType', None]: 1516 ### A fully-imported module resolves from `sys.modules` regardless of the active venv, 1517 ### so skip activating the venv (which resolves and creates its directories). 1518 import sys 1519 loaded_module = sys.modules.get(_name, None) 1520 if ( 1521 loaded_module is not None 1522 and not getattr(getattr(loaded_module, '__spec__', None), '_initializing', False) 1523 ): 1524 return loaded_module 1525 1526 with Venv(venv=venv, debug=debug): 1527 ### determine the import method (lazy vs normal) 1528 from meerschaum.utils.misc import filter_keywords 1529 import_method = ( 1530 _import_module if not lazy 1531 else lazy_import 1532 ) 1533 try: 1534 mod = import_method(_name, **(filter_keywords(import_method, **kw))) 1535 except Exception as e: 1536 if warn: 1537 import traceback 1538 traceback.print_exception(type(e), e, e.__traceback__) 1539 warn_function( 1540 f"Failed to import module '{_name}'.\nException:\n{e}", 1541 ImportWarning, 1542 stacklevel = (5 if lazy else 4), 1543 color = False, 1544 ) 1545 mod = None 1546 return mod 1547 1548 modules = [] 1549 for name in names: 1550 ### Check if package is a declared dependency. 1551 root_name = name.split('.')[0] if split else name 1552 install_name = _import_to_install_name(root_name) 1553 1554 if install_name is None: 1555 install_name = root_name 1556 if warn and root_name != 'plugins': 1557 warn_function( 1558 f"Package '{root_name}' is not declared in meerschaum.utils.packages.", 1559 ImportWarning, 1560 stacklevel = 3, 1561 color = False 1562 ) 1563 1564 ### Determine if the package exists. 1565 if precheck is False: 1566 found_module = ( 1567 do_import( 1568 name, debug=debug, warn=False, venv=venv, color=color, 1569 check_update=False, check_pypi=False, split=split, 1570 ) is not None 1571 ) 1572 else: 1573 installed_cache_key = (name, venv, split, allow_outside_venv) 1574 if check_is_installed: 1575 with _locks['_is_installed_first_check']: 1576 if not _is_installed_first_check.get(installed_cache_key, False): 1577 package_is_installed = is_installed( 1578 name, 1579 venv = venv, 1580 split = split, 1581 allow_outside_venv = allow_outside_venv, 1582 debug = debug, 1583 ) 1584 if package_is_installed: 1585 _is_installed_first_check[installed_cache_key] = True 1586 else: 1587 package_is_installed = True 1588 else: 1589 package_is_installed = venv_contains_package( 1590 name, 1591 venv=venv, 1592 split=split, 1593 debug=debug, 1594 ) 1595 found_module = package_is_installed 1596 1597 if not found_module: 1598 if install: 1599 with _locks['emitted_auto_install_warning']: 1600 if warn and not emitted_auto_install_warning: 1601 emitted_auto_install_warning = True 1602 warn_function( 1603 "Meerschaum is installing a missing runtime dependency. " 1604 + f"Set {no_auto_install_env_var}=1 to disable automatic downloads.", 1605 ImportWarning, 1606 stacklevel=3, 1607 color=False, 1608 ) 1609 if not pip_install( 1610 install_name, 1611 venv = venv, 1612 split = False, 1613 check_update = check_update, 1614 color = color, 1615 debug = debug 1616 ) and warn: 1617 warn_function( 1618 f"Failed to install '{install_name}'.", 1619 ImportWarning, 1620 stacklevel = 3, 1621 color = False, 1622 ) 1623 elif warn: 1624 ### Raise a warning if we can't find the package and install = False. 1625 warn_function( 1626 (f"\n\nMissing package '{name}' from virtual environment '{venv}'; " 1627 + "some features will not work correctly." 1628 + ( 1629 f"\n\nUnset {no_auto_install_env_var} to allow package installation.\n" 1630 if no_auto_install_env_var in os.environ 1631 else "\n\nSet install=True when calling attempt_import.\n" 1632 )), 1633 ImportWarning, 1634 stacklevel = 3, 1635 color = False, 1636 ) 1637 1638 ### Do the import. Will be lazy if lazy=True. 1639 m = do_import( 1640 name, debug=debug, warn=warn, venv=venv, color=color, 1641 check_update=check_update, check_pypi=check_pypi, install=install, split=split, 1642 ) 1643 modules.append(m) 1644 1645 modules = tuple(modules) 1646 if len(modules) == 1: 1647 return modules[0] 1648 return modules
Raise a warning if packages are not installed; otherwise import and return modules.
If lazy is True, return lazy-imported modules.
Returns tuple of modules if multiple names are provided, else returns one module.
Parameters
- names (List[str]): The packages to be imported.
- lazy (bool, default True):
If
True, lazily load packages. - warn (bool, default True):
If
True, raise a warning if a package cannot be imported. - install (bool, default True):
If
True, attempt to install a missing package into the designated virtual environment. Ifcheck_updateis True, install updates if available. - venv (Optional[str], default 'mrsm'): The virtual environment in which to search for packages and to install packages into.
- precheck (bool, default True):
If
True, attempt to find module before importing (necessary for checking if modules exist and retaining lazy imports), otherwise assume lazy isFalse. - split (bool, default True):
If
True, split packages' names on'.'. - check_update (bool, default False):
If
TrueandinstallisTrue, install updates if the required minimum version does not match. - check_pypi (bool, default False):
If
Trueandcheck_updateisTrue, check PyPI when determining whether an update is required. - check_is_installed (bool, default True):
If
True, check if the package is contained in the virtual environment. - allow_outside_venv (bool, default True):
If
True, search outside of the specified virtual environment if the package cannot be found. Setting toFalsewill reinstall the package into a virtual environment, even if it is installed outside. - color (bool, default True):
If
False, do not print ANSI colors.
Returns
- The specified modules. If they're not available and
installisTrue, it will first - download them into a virtual environment and return the modules.
Examples
>>> pandas, sqlalchemy = attempt_import('pandas', 'sqlalchemy')
>>> pandas = attempt_import('pandas')
22class Connector(metaclass=abc.ABCMeta): 23 """ 24 The base connector class to hold connection attributes. 25 """ 26 27 IS_INSTANCE: bool = False 28 29 def __init__( 30 self, 31 type: Optional[str] = None, 32 label: Optional[str] = None, 33 **kw: Any 34 ): 35 """ 36 Set the given keyword arguments as attributes. 37 38 Parameters 39 ---------- 40 type: str 41 The `type` of the connector (e.g. `sql`, `api`, `plugin`). 42 43 label: str 44 The `label` for the connector. 45 46 47 Examples 48 -------- 49 Run `mrsm edit config` and to edit connectors in the YAML file: 50 51 ```yaml 52 meerschaum: 53 connections: 54 {type}: 55 {label}: 56 ### attributes go here 57 ``` 58 59 """ 60 self._original_dict = copy.deepcopy(self.__dict__) 61 self._set_attributes(type=type, label=label, **kw) 62 63 ### NOTE: Override `REQUIRED_ATTRIBUTES` if `uri` is set. 64 self.verify_attributes( 65 ['uri'] 66 if 'uri' in self.__dict__ 67 else getattr(self, 'REQUIRED_ATTRIBUTES', None) 68 ) 69 70 def _reset_attributes(self): 71 self.__dict__ = self._original_dict 72 73 def _set_attributes( 74 self, 75 *args, 76 inherit_default: bool = True, 77 **kw: Any 78 ): 79 from meerschaum._internal.static import STATIC_CONFIG 80 from meerschaum.utils.warnings import error 81 82 self._attributes = {} 83 84 default_label = STATIC_CONFIG['connectors']['default_label'] 85 86 ### NOTE: Support the legacy method of explicitly passing the type. 87 label = kw.get('label', None) 88 if label is None: 89 if len(args) == 2: 90 label = args[1] 91 elif len(args) == 0: 92 label = None 93 else: 94 label = args[0] 95 96 if label == 'default': 97 error( 98 f"Label cannot be 'default'. Did you mean '{default_label}'?", 99 InvalidAttributesError, 100 ) 101 self.__dict__['label'] = label 102 103 from meerschaum.config import get_config 104 conn_configs = copy.deepcopy(get_config('meerschaum', 'connectors')) 105 connector_config = copy.deepcopy(get_config('system', 'connectors')) 106 107 ### inherit attributes from 'default' if exists 108 if inherit_default: 109 inherit_from = 'default' 110 if self.type in conn_configs and inherit_from in conn_configs[self.type]: 111 _inherit_dict = copy.deepcopy(conn_configs[self.type][inherit_from]) 112 self._attributes.update(_inherit_dict) 113 114 ### load user config into self._attributes 115 if self.type in conn_configs and self.label in conn_configs[self.type]: 116 self._attributes.update(conn_configs[self.type][self.label] or {}) 117 118 ### load system config into self._sys_config 119 ### (deep copy so future Connectors don't inherit changes) 120 if self.type in connector_config: 121 self._sys_config = copy.deepcopy(connector_config[self.type]) 122 123 ### add additional arguments or override configuration 124 self._attributes.update(kw) 125 126 ### finally, update __dict__ with _attributes. 127 self.__dict__.update(self._attributes) 128 129 def verify_attributes( 130 self, 131 required_attributes: Optional[List[str]] = None, 132 debug: bool = False, 133 ) -> None: 134 """ 135 Ensure that the required attributes have been met. 136 137 The Connector base class checks the minimum requirements. 138 Child classes may enforce additional requirements. 139 140 Parameters 141 ---------- 142 required_attributes: Optional[List[str]], default None 143 Attributes to be verified. If `None`, default to `['label']`. 144 145 debug: bool, default False 146 Verbosity toggle. 147 148 Returns 149 ------- 150 Don't return anything. 151 152 Raises 153 ------ 154 An error if any of the required attributes are missing. 155 """ 156 from meerschaum.utils.warnings import error 157 from meerschaum.utils.misc import items_str 158 if required_attributes is None: 159 required_attributes = ['type', 'label'] 160 161 missing_attributes = set() 162 for a in required_attributes: 163 if a not in self.__dict__: 164 missing_attributes.add(a) 165 if len(missing_attributes) > 0: 166 error( 167 ( 168 f"Missing {items_str(list(missing_attributes))} " 169 + f"for connector '{self.type}:{self.label}'." 170 ), 171 InvalidAttributesError, 172 silent=True, 173 stack=False 174 ) 175 176 177 def __str__(self): 178 """ 179 When cast to a string, return type:label. 180 """ 181 return f"{self.type}:{self.label}" 182 183 def __repr__(self): 184 """ 185 Represent the connector as type:label. 186 """ 187 return str(self) 188 189 @property 190 def meta(self) -> Dict[str, Any]: 191 """ 192 Return the keys needed to reconstruct this Connector. 193 """ 194 _meta = { 195 key: value 196 for key, value in self.__dict__.items() 197 if not str(key).startswith('_') 198 } 199 _meta.update({ 200 'type': self.type, 201 'label': self.label, 202 }) 203 return _meta 204 205 206 @property 207 def type(self) -> str: 208 """ 209 Return the type for this connector. 210 """ 211 _type = self.__dict__.get('type', None) 212 if _type is None: 213 import re 214 is_executor = self.__class__.__name__.lower().endswith('executor') 215 suffix_regex = ( 216 r'connector$' 217 if not is_executor 218 else r'executor$' 219 ) 220 _type = re.sub(suffix_regex, '', self.__class__.__name__.lower()) 221 if not _type or _type.lower() == 'instance': 222 raise ValueError("No type could be determined for this connector.") 223 self.__dict__['type'] = _type 224 return _type 225 226 227 @property 228 def label(self) -> str: 229 """ 230 Return the label for this connector. 231 """ 232 _label = self.__dict__.get('label', None) 233 if _label is None: 234 from meerschaum._internal.static import STATIC_CONFIG 235 _label = STATIC_CONFIG['connectors']['default_label'] 236 self.__dict__['label'] = _label 237 return _label
The base connector class to hold connection attributes.
29 def __init__( 30 self, 31 type: Optional[str] = None, 32 label: Optional[str] = None, 33 **kw: Any 34 ): 35 """ 36 Set the given keyword arguments as attributes. 37 38 Parameters 39 ---------- 40 type: str 41 The `type` of the connector (e.g. `sql`, `api`, `plugin`). 42 43 label: str 44 The `label` for the connector. 45 46 47 Examples 48 -------- 49 Run `mrsm edit config` and to edit connectors in the YAML file: 50 51 ```yaml 52 meerschaum: 53 connections: 54 {type}: 55 {label}: 56 ### attributes go here 57 ``` 58 59 """ 60 self._original_dict = copy.deepcopy(self.__dict__) 61 self._set_attributes(type=type, label=label, **kw) 62 63 ### NOTE: Override `REQUIRED_ATTRIBUTES` if `uri` is set. 64 self.verify_attributes( 65 ['uri'] 66 if 'uri' in self.__dict__ 67 else getattr(self, 'REQUIRED_ATTRIBUTES', None) 68 )
129 def verify_attributes( 130 self, 131 required_attributes: Optional[List[str]] = None, 132 debug: bool = False, 133 ) -> None: 134 """ 135 Ensure that the required attributes have been met. 136 137 The Connector base class checks the minimum requirements. 138 Child classes may enforce additional requirements. 139 140 Parameters 141 ---------- 142 required_attributes: Optional[List[str]], default None 143 Attributes to be verified. If `None`, default to `['label']`. 144 145 debug: bool, default False 146 Verbosity toggle. 147 148 Returns 149 ------- 150 Don't return anything. 151 152 Raises 153 ------ 154 An error if any of the required attributes are missing. 155 """ 156 from meerschaum.utils.warnings import error 157 from meerschaum.utils.misc import items_str 158 if required_attributes is None: 159 required_attributes = ['type', 'label'] 160 161 missing_attributes = set() 162 for a in required_attributes: 163 if a not in self.__dict__: 164 missing_attributes.add(a) 165 if len(missing_attributes) > 0: 166 error( 167 ( 168 f"Missing {items_str(list(missing_attributes))} " 169 + f"for connector '{self.type}:{self.label}'." 170 ), 171 InvalidAttributesError, 172 silent=True, 173 stack=False 174 )
Ensure that the required attributes have been met.
The Connector base class checks the minimum requirements. Child classes may enforce additional requirements.
Parameters
- required_attributes (Optional[List[str]], default None):
Attributes to be verified. If
None, default to['label']. - debug (bool, default False): Verbosity toggle.
Returns
- Don't return anything.
Raises
- An error if any of the required attributes are missing.
189 @property 190 def meta(self) -> Dict[str, Any]: 191 """ 192 Return the keys needed to reconstruct this Connector. 193 """ 194 _meta = { 195 key: value 196 for key, value in self.__dict__.items() 197 if not str(key).startswith('_') 198 } 199 _meta.update({ 200 'type': self.type, 201 'label': self.label, 202 }) 203 return _meta
Return the keys needed to reconstruct this Connector.
206 @property 207 def type(self) -> str: 208 """ 209 Return the type for this connector. 210 """ 211 _type = self.__dict__.get('type', None) 212 if _type is None: 213 import re 214 is_executor = self.__class__.__name__.lower().endswith('executor') 215 suffix_regex = ( 216 r'connector$' 217 if not is_executor 218 else r'executor$' 219 ) 220 _type = re.sub(suffix_regex, '', self.__class__.__name__.lower()) 221 if not _type or _type.lower() == 'instance': 222 raise ValueError("No type could be determined for this connector.") 223 self.__dict__['type'] = _type 224 return _type
Return the type for this connector.
227 @property 228 def label(self) -> str: 229 """ 230 Return the label for this connector. 231 """ 232 _label = self.__dict__.get('label', None) 233 if _label is None: 234 from meerschaum._internal.static import STATIC_CONFIG 235 _label = STATIC_CONFIG['connectors']['default_label'] 236 self.__dict__['label'] = _label 237 return _label
Return the label for this connector.
18class InstanceConnector(Connector): 19 """ 20 Instance connectors define the interface for managing pipes and provide methods 21 for management of users, plugins, tokens, and other metadata built atop pipes. 22 """ 23 24 IS_INSTANCE: bool = True 25 IS_THREAD_SAFE: bool = False 26 27 from ._users import ( 28 get_users_pipe, 29 register_user, 30 get_user_id, 31 get_username, 32 get_users, 33 edit_user, 34 delete_user, 35 get_user_password_hash, 36 get_user_type, 37 get_user_attributes, 38 ) 39 40 from ._plugins import ( 41 get_plugins_pipe, 42 register_plugin, 43 get_plugin_user_id, 44 delete_plugin, 45 get_plugin_id, 46 get_plugin_version, 47 get_plugins, 48 get_plugin_user_id, 49 get_plugin_username, 50 get_plugin_attributes, 51 ) 52 53 from ._tokens import ( 54 get_tokens_pipe, 55 register_token, 56 edit_token, 57 invalidate_token, 58 delete_token, 59 get_token, 60 get_tokens, 61 get_token_model, 62 get_token_secret_hash, 63 token_exists, 64 get_token_scopes, 65 ) 66 67 from ._pipes import ( 68 register_pipe, 69 get_pipe_attributes, 70 get_pipe_id, 71 edit_pipe, 72 delete_pipe, 73 fetch_pipes_keys, 74 pipe_exists, 75 drop_pipe, 76 drop_pipe_indices, 77 sync_pipe, 78 create_pipe_indices, 79 clear_pipe, 80 get_pipe_data, 81 get_pipe_docs, 82 get_sync_time, 83 get_pipe_columns_types, 84 get_pipe_columns_indices, 85 get_pipe_size, 86 compress_pipe, 87 decompress_pipe, 88 vacuum_pipe, 89 analyze_pipe, 90 partition_pipe, 91 )
Instance connectors define the interface for managing pipes and provide methods for management of users, plugins, tokens, and other metadata built atop pipes.
18def get_users_pipe(self) -> 'mrsm.Pipe': 19 """ 20 Return the pipe used for users registration. 21 """ 22 if '_users_pipe' in self.__dict__: 23 return self._users_pipe 24 25 cache_connector = self.__dict__.get('_cache_connector', None) 26 self._users_pipe = mrsm.Pipe( 27 'mrsm', 'users', 28 instance=self, 29 target='mrsm_users', 30 temporary=True, 31 cache=True, 32 cache_connector_keys=cache_connector, 33 static=True, 34 null_indices=False, 35 columns={ 36 'primary': 'user_id', 37 }, 38 dtypes={ 39 'user_id': 'uuid', 40 'username': 'string', 41 'password_hash': 'string', 42 'email': 'string', 43 'user_type': 'string', 44 'attributes': 'json', 45 }, 46 indices={ 47 'unique': 'username', 48 }, 49 ) 50 return self._users_pipe
Return the pipe used for users registration.
53def register_user( 54 self, 55 user: User, 56 debug: bool = False, 57 **kwargs: Any 58) -> mrsm.SuccessTuple: 59 """ 60 Register a new user to the users pipe. 61 """ 62 users_pipe = self.get_users_pipe() 63 user.user_id = uuid.uuid4() 64 sync_success, sync_msg = users_pipe.sync( 65 [{ 66 'user_id': user.user_id, 67 'username': user.username, 68 'email': user.email, 69 'password_hash': user.password_hash, 70 'user_type': user.type, 71 'attributes': user.attributes, 72 }], 73 check_existing=False, 74 debug=debug, 75 ) 76 if not sync_success: 77 return False, f"Failed to register user '{user.username}':\n{sync_msg}" 78 79 return True, "Success"
Register a new user to the users pipe.
82def get_user_id(self, user: User, debug: bool = False) -> Union[uuid.UUID, None]: 83 """ 84 Return a user's ID from the username. 85 """ 86 users_pipe = self.get_users_pipe() 87 result_df = users_pipe.get_data(['user_id'], params={'username': user.username}, limit=1) 88 if result_df is None or len(result_df) == 0: 89 return None 90 return result_df['user_id'][0]
Return a user's ID from the username.
93def get_username(self, user_id: Any, debug: bool = False) -> Any: 94 """ 95 Return the username from the given ID. 96 """ 97 users_pipe = self.get_users_pipe() 98 return users_pipe.get_value('username', {'user_id': user_id}, debug=debug)
Return the username from the given ID.
101def get_users( 102 self, 103 debug: bool = False, 104 **kw: Any 105) -> List[str]: 106 """ 107 Get the registered usernames. 108 """ 109 users_pipe = self.get_users_pipe() 110 df = users_pipe.get_data() 111 if df is None: 112 return [] 113 114 return list(df['username'])
Get the registered usernames.
117def edit_user(self, user: User, debug: bool = False) -> mrsm.SuccessTuple: 118 """ 119 Edit the attributes for an existing user. 120 """ 121 users_pipe = self.get_users_pipe() 122 user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug) 123 124 doc = {'user_id': user_id} 125 if user.email != '': 126 doc['email'] = user.email 127 if user.password_hash != '': 128 doc['password_hash'] = user.password_hash 129 if user.type != '': 130 doc['user_type'] = user.type 131 if user.attributes: 132 doc['attributes'] = user.attributes 133 134 sync_success, sync_msg = users_pipe.sync([doc], debug=debug) 135 if not sync_success: 136 return False, f"Failed to edit user '{user.username}':\n{sync_msg}" 137 138 return True, "Success"
Edit the attributes for an existing user.
141def delete_user(self, user: User, debug: bool = False) -> mrsm.SuccessTuple: 142 """ 143 Delete a user from the users table. 144 """ 145 user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug) 146 users_pipe = self.get_users_pipe() 147 clear_success, clear_msg = users_pipe.clear(params={'user_id': user_id}, debug=debug) 148 if not clear_success: 149 return False, f"Failed to delete user '{user}':\n{clear_msg}" 150 return True, "Success"
Delete a user from the users table.
153def get_user_password_hash(self, user: User, debug: bool = False) -> Union[uuid.UUID, None]: 154 """ 155 Get a user's password hash from the users table. 156 """ 157 user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug) 158 users_pipe = self.get_users_pipe() 159 result_df = users_pipe.get_data(['password_hash'], params={'user_id': user_id}, debug=debug) 160 if result_df is None or len(result_df) == 0: 161 return None 162 163 return result_df['password_hash'][0]
Get a user's password hash from the users table.
166def get_user_type(self, user: User, debug: bool = False) -> Union[str, None]: 167 """ 168 Get a user's type from the users table. 169 """ 170 user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug) 171 users_pipe = self.get_users_pipe() 172 result_df = users_pipe.get_data(['user_type'], params={'user_id': user_id}, debug=debug) 173 if result_df is None or len(result_df) == 0: 174 return None 175 176 return result_df['user_type'][0]
Get a user's type from the users table.
179def get_user_attributes(self, user: User, debug: bool = False) -> Union[Dict[str, Any], None]: 180 """ 181 Get a user's attributes from the users table. 182 """ 183 user_id = user.user_id if user.user_id is not None else self.get_user_id(user, debug=debug) 184 users_pipe = self.get_users_pipe() 185 result_df = users_pipe.get_data(['attributes'], params={'user_id': user_id}, debug=debug) 186 if result_df is None or len(result_df) == 0: 187 return None 188 189 return result_df['attributes'][0]
Get a user's attributes from the users table.
16def get_plugins_pipe(self) -> 'mrsm.Pipe': 17 """ 18 Return the internal pipe for syncing plugins metadata. 19 """ 20 if '_plugins_pipe' in self.__dict__: 21 return self._plugins_pipe 22 23 cache_connector = self.__dict__.get('_cache_connector', None) 24 users_pipe = self.get_users_pipe() 25 user_id_dtype = users_pipe.dtypes.get('user_id', 'uuid') 26 27 self._plugins_pipe = mrsm.Pipe( 28 'mrsm', 'plugins', 29 instance=self, 30 target='mrsm_plugins', 31 temporary=True, 32 cache=True, 33 cache_connector_keys=cache_connector, 34 static=True, 35 null_indices=False, 36 columns={ 37 'primary': 'plugin_name', 38 'user_id': 'user_id', 39 }, 40 dtypes={ 41 'plugin_name': 'string', 42 'user_id': user_id_dtype, 43 'attributes': 'json', 44 'version': 'string', 45 }, 46 ) 47 return self._plugins_pipe
Return the internal pipe for syncing plugins metadata.
50def register_plugin(self, plugin: Plugin, debug: bool = False) -> mrsm.SuccessTuple: 51 """ 52 Register a new plugin to the plugins table. 53 """ 54 plugins_pipe = self.get_plugins_pipe() 55 users_pipe = self.get_users_pipe() 56 user_id = self.get_plugin_user_id(plugin) 57 if user_id is not None: 58 username = self.get_username(user_id, debug=debug) 59 return False, f"{plugin} is already registered to '{username}'." 60 61 doc = { 62 'plugin_name': plugin.name, 63 'version': plugin.version, 64 'attributes': plugin.attributes, 65 'user_id': plugin.user_id, 66 } 67 68 sync_success, sync_msg = plugins_pipe.sync( 69 [doc], 70 check_existing=False, 71 debug=debug, 72 ) 73 if not sync_success: 74 return False, f"Failed to register {plugin}:\n{sync_msg}" 75 76 return True, "Success"
Register a new plugin to the plugins table.
79def get_plugin_user_id(self, plugin: Plugin, debug: bool = False) -> Union[uuid.UUID, None]: 80 """ 81 Return the user ID for plugin's owner. 82 """ 83 plugins_pipe = self.get_plugins_pipe() 84 return plugins_pipe.get_value('user_id', {'plugin_name': plugin.name}, debug=debug)
Return the user ID for plugin's owner.
105def delete_plugin(self, plugin: Plugin, debug: bool = False) -> mrsm.SuccessTuple: 106 """ 107 Delete a plugin's registration. 108 """ 109 plugin_id = self.get_plugin_id(plugin, debug=debug) 110 if plugin_id is None: 111 return False, f"{plugin} is not registered." 112 113 plugins_pipe = self.get_plugins_pipe() 114 clear_success, clear_msg = plugins_pipe.clear(params={'plugin_name': plugin.name}, debug=debug) 115 if not clear_success: 116 return False, f"Failed to delete {plugin}:\n{clear_msg}" 117 return True, "Success"
Delete a plugin's registration.
97def get_plugin_id(self, plugin: Plugin, debug: bool = False) -> Union[str, None]: 98 """ 99 Return a plugin's ID. 100 """ 101 user_id = self.get_plugin_user_id(plugin, debug=debug) 102 return plugin.name if user_id is not None else None
Return a plugin's ID.
120def get_plugin_version(self, plugin: Plugin, debug: bool = False) -> Union[str, None]: 121 """ 122 Return the version for a plugin. 123 """ 124 plugins_pipe = self.get_plugins_pipe() 125 return plugins_pipe.get_value('version', {'plugin_name': plugin.name}, debug=debug)
Return the version for a plugin.
136def get_plugins( 137 self, 138 user_id: Optional[int] = None, 139 search_term: Optional[str] = None, 140 debug: bool = False, 141 **kw: Any 142) -> List[str]: 143 """ 144 Return a list of plugin names. 145 """ 146 plugins_pipe = self.get_plugins_pipe() 147 params = {} 148 if user_id: 149 params['user_id'] = user_id 150 151 df = plugins_pipe.get_data(['plugin_name'], params=params, debug=debug) 152 if df is None: 153 return [] 154 155 docs = df.to_dict(orient='records') 156 return [ 157 plugin_name 158 for doc in docs 159 if (plugin_name := doc['plugin_name']).startswith(search_term or '') 160 ]
Return a list of plugin names.
87def get_plugin_username(self, plugin: Plugin, debug: bool = False) -> Union[uuid.UUID, None]: 88 """ 89 Return the username for plugin's owner. 90 """ 91 user_id = self.get_plugin_user_id(plugin, debug=debug) 92 if user_id is None: 93 return None 94 return self.get_username(user_id, debug=debug)
Return the username for plugin's owner.
128def get_plugin_attributes(self, plugin: Plugin, debug: bool = False) -> Dict[str, Any]: 129 """ 130 Return the attributes for a plugin. 131 """ 132 plugins_pipe = self.get_plugins_pipe() 133 return plugins_pipe.get_value('attributes', {'plugin_name': plugin.name}, debug=debug) or {}
Return the attributes for a plugin.
22def get_tokens_pipe(self) -> mrsm.Pipe: 23 """ 24 Return the internal pipe for tokens management. 25 """ 26 if '_tokens_pipe' in self.__dict__: 27 return self._tokens_pipe 28 29 users_pipe = self.get_users_pipe() 30 user_id_dtype = ( 31 users_pipe._attributes.get('parameters', {}).get('dtypes', {}).get('user_id', 'uuid') 32 ) 33 34 cache_connector = self.__dict__.get('_cache_connector', None) 35 36 self._tokens_pipe = mrsm.Pipe( 37 'mrsm', 'tokens', 38 instance=self, 39 target='mrsm_tokens', 40 temporary=True, 41 cache=True, 42 cache_connector_keys=cache_connector, 43 static=True, 44 autotime=True, 45 null_indices=False, 46 columns={ 47 'datetime': 'creation', 48 'primary': 'id', 49 }, 50 indices={ 51 'unique': 'label', 52 'user_id': 'user_id', 53 }, 54 dtypes={ 55 'id': 'uuid', 56 'creation': 'datetime', 57 'expiration': 'datetime', 58 'is_valid': 'bool', 59 'label': 'string', 60 'user_id': user_id_dtype, 61 'scopes': 'json', 62 'secret_hash': 'string', 63 }, 64 ) 65 return self._tokens_pipe
Return the internal pipe for tokens management.
68def register_token( 69 self, 70 token: Token, 71 debug: bool = False, 72) -> mrsm.SuccessTuple: 73 """ 74 Register the new token to the tokens table. 75 """ 76 token_id, token_secret = token.generate_credentials() 77 tokens_pipe = self.get_tokens_pipe() 78 user_id = self.get_user_id(token.user) if token.user is not None else None 79 if user_id is None: 80 return False, "Cannot register a token without a user." 81 82 doc = { 83 'id': token_id, 84 'user_id': user_id, 85 'creation': datetime.now(timezone.utc), 86 'expiration': token.expiration, 87 'label': token.label, 88 'is_valid': token.is_valid, 89 'scopes': list(token.scopes) if token.scopes else [], 90 'secret_hash': hash_password( 91 str(token_secret), 92 rounds=STATIC_CONFIG['tokens']['hash_rounds'] 93 ), 94 } 95 sync_success, sync_msg = tokens_pipe.sync([doc], check_existing=False, debug=debug) 96 if not sync_success: 97 return False, f"Failed to register token:\n{sync_msg}" 98 return True, "Success"
Register the new token to the tokens table.
101def edit_token(self, token: Token, debug: bool = False) -> mrsm.SuccessTuple: 102 """ 103 Persist the token's in-memory state to the tokens pipe. 104 """ 105 if not token.id: 106 return False, "Token ID is not set." 107 108 if not token.exists(debug=debug): 109 return False, f"Token {token.id} does not exist." 110 111 if not token.creation: 112 token_model = self.get_token_model(token.id) 113 token.creation = token_model.creation 114 115 tokens_pipe = self.get_tokens_pipe() 116 doc = { 117 'id': token.id, 118 'creation': token.creation, 119 'expiration': token.expiration, 120 'label': token.label, 121 'is_valid': token.is_valid, 122 'scopes': list(token.scopes) if token.scopes else [], 123 } 124 sync_success, sync_msg = tokens_pipe.sync([doc], debug=debug) 125 if not sync_success: 126 return False, f"Failed to edit token '{token.id}':\n{sync_msg}" 127 128 return True, "Success"
Persist the token's in-memory state to the tokens pipe.
131def invalidate_token(self, token: Token, debug: bool = False) -> mrsm.SuccessTuple: 132 """ 133 Set `is_valid` to `False` for the given token. 134 """ 135 if not token.id: 136 return False, "Token ID is not set." 137 138 if not token.exists(debug=debug): 139 return False, f"Token {token.id} does not exist." 140 141 if not token.creation: 142 token_model = self.get_token_model(token.id) 143 token.creation = token_model.creation 144 145 token.is_valid = False 146 tokens_pipe = self.get_tokens_pipe() 147 doc = { 148 'id': token.id, 149 'creation': token.creation, 150 'is_valid': False, 151 } 152 sync_success, sync_msg = tokens_pipe.sync([doc], debug=debug) 153 if not sync_success: 154 return False, f"Failed to invalidate token '{token.id}':\n{sync_msg}" 155 156 return True, "Success"
Set is_valid to False for the given token.
159def delete_token(self, token: Token, debug: bool = False) -> mrsm.SuccessTuple: 160 """ 161 Delete the given token from the tokens table. 162 """ 163 if not token.id: 164 return False, "Token ID is not set." 165 166 if not token.exists(debug=debug): 167 return False, f"Token {token.id} does not exist." 168 169 if not token.creation: 170 token_model = self.get_token_model(token.id) 171 token.creation = token_model.creation 172 173 token.is_valid = False 174 tokens_pipe = self.get_tokens_pipe() 175 clear_success, clear_msg = tokens_pipe.clear(params={'id': token.id}, debug=debug) 176 if not clear_success: 177 return False, f"Failed to delete token '{token.id}':\n{clear_msg}" 178 179 return True, "Success"
Delete the given token from the tokens table.
235def get_token(self, token_id: Union[uuid.UUID, str], debug: bool = False) -> Union[Token, None]: 236 """ 237 Return the `Token` from its ID. 238 """ 239 from meerschaum.utils.misc import is_uuid 240 if isinstance(token_id, str): 241 if is_uuid(token_id): 242 token_id = uuid.UUID(token_id) 243 else: 244 raise ValueError("Invalid token ID.") 245 token_model = self.get_token_model(token_id) 246 if token_model is None: 247 return None 248 return Token(**dict(token_model))
Return the Token from its ID.
182def get_tokens( 183 self, 184 user: Optional[User] = None, 185 labels: Optional[List[str]] = None, 186 ids: Optional[List[uuid.UUID]] = None, 187 debug: bool = False, 188) -> List[Token]: 189 """ 190 Return a list of `Token` objects. 191 """ 192 tokens_pipe = self.get_tokens_pipe() 193 user_id = ( 194 self.get_user_id(user, debug=debug) 195 if user is not None 196 else None 197 ) 198 user_type = self.get_user_type(user, debug=debug) if user is not None else None 199 params = ( 200 { 201 'user_id': ( 202 user_id 203 if user_type != 'admin' 204 else [user_id, None] 205 ) 206 } 207 if user_id is not None 208 else {} 209 ) 210 if labels: 211 params['label'] = labels 212 if ids: 213 params['id'] = ids 214 215 if debug: 216 dprint(f"Getting tokens with {user_id=}, {params=}") 217 218 tokens_df = tokens_pipe.get_data(params=params, debug=debug) 219 if tokens_df is None: 220 return [] 221 222 if debug: 223 dprint(f"Retrieved tokens dataframe:\n{tokens_df}") 224 225 tokens_docs = tokens_df.to_dict(orient='records') 226 return [ 227 Token( 228 instance=self, 229 **token_doc 230 ) 231 for token_doc in reversed(tokens_docs) 232 ]
Return a list of Token objects.
251def get_token_model(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> 'Union[TokenModel, None]': 252 """ 253 Return a token's model from the instance. 254 """ 255 from meerschaum.models import TokenModel 256 if isinstance(token_id, Token): 257 token_id = Token.id 258 if not token_id: 259 raise ValueError("Invalid token ID.") 260 tokens_pipe = self.get_tokens_pipe() 261 doc = tokens_pipe.get_doc( 262 params={'id': token_id}, 263 debug=debug, 264 ) 265 if doc is None: 266 return None 267 return TokenModel(**doc)
Return a token's model from the instance.
270def get_token_secret_hash(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> Union[str, None]: 271 """ 272 Return the secret hash for a given token. 273 """ 274 if isinstance(token_id, Token): 275 token_id = token_id.id 276 if not token_id: 277 raise ValueError("Invalid token ID.") 278 tokens_pipe = self.get_tokens_pipe() 279 return tokens_pipe.get_value('secret_hash', params={'id': token_id}, debug=debug)
Return the secret hash for a given token.
308def token_exists(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> bool: 309 """ 310 Return `True` if a token exists in the tokens pipe. 311 """ 312 if isinstance(token_id, Token): 313 token_id = token_id.id 314 if not token_id: 315 raise ValueError("Invalid token ID.") 316 317 tokens_pipe = self.get_tokens_pipe() 318 return tokens_pipe.get_value('creation', params={'id': token_id}, debug=debug) is not None
Return True if a token exists in the tokens pipe.
295def get_token_scopes(self, token_id: Union[uuid.UUID, Token], debug: bool = False) -> List[str]: 296 """ 297 Return the scopes for a token. 298 """ 299 if isinstance(token_id, Token): 300 token_id = token_id.id 301 if not token_id: 302 raise ValueError("Invalid token ID.") 303 304 tokens_pipe = self.get_tokens_pipe() 305 return tokens_pipe.get_value('scopes', params={'id': token_id}, debug=debug) or []
Return the scopes for a token.
17@abc.abstractmethod 18def register_pipe( 19 self, 20 pipe: mrsm.Pipe, 21 debug: bool = False, 22 **kwargs: Any 23) -> mrsm.SuccessTuple: 24 """ 25 Insert the pipe's attributes into the internal `pipes` table. 26 27 Parameters 28 ---------- 29 pipe: mrsm.Pipe 30 The pipe to be registered. 31 32 Returns 33 ------- 34 A `SuccessTuple` of the result. 35 """
Insert the pipe's attributes into the internal pipes table.
Parameters
- pipe (mrsm.Pipe): The pipe to be registered.
Returns
- A
SuccessTupleof the result.
37@abc.abstractmethod 38def get_pipe_attributes( 39 self, 40 pipe: mrsm.Pipe, 41 debug: bool = False, 42 **kwargs: Any 43) -> Dict[str, Any]: 44 """ 45 Return the pipe's document from the internal `pipes` table. 46 47 Parameters 48 ---------- 49 pipe: mrsm.Pipe 50 The pipe whose attributes should be retrieved. 51 52 Returns 53 ------- 54 The document that matches the keys of the pipe. 55 """
Return the pipe's document from the internal pipes table.
Parameters
- pipe (mrsm.Pipe): The pipe whose attributes should be retrieved.
Returns
- The document that matches the keys of the pipe.
57@abc.abstractmethod 58def get_pipe_id( 59 self, 60 pipe: mrsm.Pipe, 61 debug: bool = False, 62 **kwargs: Any 63) -> Union[str, int, None]: 64 """ 65 Return the `id` for the pipe if it exists. 66 67 Parameters 68 ---------- 69 pipe: mrsm.Pipe 70 The pipe whose `id` to fetch. 71 72 Returns 73 ------- 74 The `id` for the pipe's document or `None`. 75 """
Return the id for the pipe if it exists.
Parameters
- pipe (mrsm.Pipe):
The pipe whose
idto fetch.
Returns
- The
idfor the pipe's document orNone.
77def edit_pipe( 78 self, 79 pipe: mrsm.Pipe, 80 debug: bool = False, 81 **kwargs: Any 82) -> mrsm.SuccessTuple: 83 """ 84 Edit the attributes of the pipe. 85 86 Parameters 87 ---------- 88 pipe: mrsm.Pipe 89 The pipe whose in-memory parameters must be persisted. 90 91 Returns 92 ------- 93 A `SuccessTuple` indicating success. 94 """ 95 raise NotImplementedError
Edit the attributes of the pipe.
Parameters
- pipe (mrsm.Pipe): The pipe whose in-memory parameters must be persisted.
Returns
- A
SuccessTupleindicating success.
97def delete_pipe( 98 self, 99 pipe: mrsm.Pipe, 100 debug: bool = False, 101 **kwargs: Any 102) -> mrsm.SuccessTuple: 103 """ 104 Delete a pipe's registration from the `pipes` collection. 105 106 Parameters 107 ---------- 108 pipe: mrsm.Pipe 109 The pipe to be deleted. 110 111 Returns 112 ------- 113 A `SuccessTuple` indicating success. 114 """ 115 raise NotImplementedError
Delete a pipe's registration from the pipes collection.
Parameters
- pipe (mrsm.Pipe): The pipe to be deleted.
Returns
- A
SuccessTupleindicating success.
117@abc.abstractmethod 118def fetch_pipes_keys( 119 self, 120 connector_keys: Optional[List[str]] = None, 121 metric_keys: Optional[List[str]] = None, 122 location_keys: Optional[List[str]] = None, 123 tags: Optional[List[str]] = None, 124 debug: bool = False, 125 **kwargs: Any 126) -> Union[ 127 List[Tuple[str, str, str]], 128 List[Tuple[str, str, str, Union[Dict[str, Any], List[str]]]], 129 Dict[Union[int, str], Tuple[str, str, str]], 130 Dict[Union[int, str], Tuple[str, str, str, Union[Dict[str, Any], List[str]]]], 131]: 132 """ 133 Return registered pipes' keys according to the provided filters. 134 135 May return either a list of key tuples or a dictionary mapping pipe IDs to key tuples. 136 When returning a dictionary, the key is the pipe's unique ID (int or str). 137 Tuples may be length 3 `(connector_keys, metric_key, location_key)` or length 4 138 with parameters or tags appended as the fourth element. 139 140 Parameters 141 ---------- 142 connector_keys: list[str] | None, default None 143 The keys passed via `-c`. 144 145 metric_keys: list[str] | None, default None 146 The keys passed via `-m`. 147 148 location_keys: list[str] | None, default None 149 The keys passed via `-l`. 150 151 tags: List[str] | None, default None 152 Tags passed via `--tags` which are stored under `parameters:tags`. 153 154 Returns 155 ------- 156 A list of tuples or a dictionary mapping pipe IDs to tuples. 157 You may return the string `"None"` for location keys in place of nulls. 158 159 Examples 160 -------- 161 >>> import meerschaum as mrsm 162 >>> conn = mrsm.get_connector('example:demo') 163 >>> 164 >>> pipe_a = mrsm.Pipe('a', 'demo', tags=['foo'], instance=conn) 165 >>> pipe_b = mrsm.Pipe('b', 'demo', tags=['bar'], instance=conn) 166 >>> pipe_a.register() 167 >>> pipe_b.register() 168 >>> 169 >>> conn.fetch_pipes_keys(['a', 'b']) 170 [('a', 'demo', 'None'), ('b', 'demo', 'None')] 171 >>> conn.fetch_pipes_keys(metric_keys=['demo']) 172 [('a', 'demo', 'None'), ('b', 'demo', 'None')] 173 >>> conn.fetch_pipes_keys(tags=['foo']) 174 [('a', 'demo', 'None')] 175 >>> conn.fetch_pipes_keys(location_keys=[None]) 176 [('a', 'demo', 'None'), ('b', 'demo', 'None')] 177 """
Return registered pipes' keys according to the provided filters.
May return either a list of key tuples or a dictionary mapping pipe IDs to key tuples.
When returning a dictionary, the key is the pipe's unique ID (int or str).
Tuples may be length 3 (connector_keys, metric_key, location_key) or length 4
with parameters or tags appended as the fourth element.
Parameters
- connector_keys (list[str] | None, default None):
The keys passed via
-c. - metric_keys (list[str] | None, default None):
The keys passed via
-m. - location_keys (list[str] | None, default None):
The keys passed via
-l. - tags (List[str] | None, default None):
Tags passed via
--tagswhich are stored underparameters:tags.
Returns
- A list of tuples or a dictionary mapping pipe IDs to tuples.
- You may return the string
"None"for location keys in place of nulls.
Examples
>>> import meerschaum as mrsm
>>> conn = mrsm.get_connector('example:demo')
>>>
>>> pipe_a = mrsm.Pipe('a', 'demo', tags=['foo'], instance=conn)
>>> pipe_b = mrsm.Pipe('b', 'demo', tags=['bar'], instance=conn)
>>> pipe_a.register()
>>> pipe_b.register()
>>>
>>> conn.fetch_pipes_keys(['a', 'b'])
[('a', 'demo', 'None'), ('b', 'demo', 'None')]
>>> conn.fetch_pipes_keys(metric_keys=['demo'])
[('a', 'demo', 'None'), ('b', 'demo', 'None')]
>>> conn.fetch_pipes_keys(tags=['foo'])
[('a', 'demo', 'None')]
>>> conn.fetch_pipes_keys(location_keys=[None])
[('a', 'demo', 'None'), ('b', 'demo', 'None')]
179@abc.abstractmethod 180def pipe_exists( 181 self, 182 pipe: mrsm.Pipe, 183 debug: bool = False, 184 **kwargs: Any 185) -> bool: 186 """ 187 Check whether a pipe's target table exists. 188 189 Parameters 190 ---------- 191 pipe: mrsm.Pipe 192 The pipe to check whether its table exists. 193 194 Returns 195 ------- 196 A `bool` indicating the table exists. 197 """
Check whether a pipe's target table exists.
Parameters
- pipe (mrsm.Pipe): The pipe to check whether its table exists.
Returns
- A
boolindicating the table exists.
199@abc.abstractmethod 200def drop_pipe( 201 self, 202 pipe: mrsm.Pipe, 203 debug: bool = False, 204 **kwargs: Any 205) -> mrsm.SuccessTuple: 206 """ 207 Drop a pipe's collection if it exists. 208 209 Parameters 210 ---------- 211 pipe: mrsm.Pipe 212 The pipe to be dropped. 213 214 Returns 215 ------- 216 A `SuccessTuple` indicating success. 217 """ 218 raise NotImplementedError
Drop a pipe's collection if it exists.
Parameters
- pipe (mrsm.Pipe): The pipe to be dropped.
Returns
- A
SuccessTupleindicating success.
220def drop_pipe_indices( 221 self, 222 pipe: mrsm.Pipe, 223 debug: bool = False, 224 **kwargs: Any 225) -> mrsm.SuccessTuple: 226 """ 227 Drop a pipe's indices. 228 229 Parameters 230 ---------- 231 pipe: mrsm.Pipe 232 The pipe whose indices need to be dropped. 233 234 Returns 235 ------- 236 A `SuccessTuple` indicating success. 237 """ 238 return False, f"Cannot drop indices for instance connectors of type '{self.type}'."
Drop a pipe's indices.
Parameters
- pipe (mrsm.Pipe): The pipe whose indices need to be dropped.
Returns
- A
SuccessTupleindicating success.
240@abc.abstractmethod 241def sync_pipe( 242 self, 243 pipe: mrsm.Pipe, 244 df: 'pd.DataFrame' = None, 245 begin: Union[datetime, int, None] = None, 246 end: Union[datetime, int, None] = None, 247 chunksize: Optional[int] = -1, 248 check_existing: bool = True, 249 debug: bool = False, 250 **kwargs: Any 251) -> mrsm.SuccessTuple: 252 """ 253 Sync a pipe using a database connection. 254 255 Parameters 256 ---------- 257 pipe: mrsm.Pipe 258 The Meerschaum Pipe instance into which to sync the data. 259 260 df: Optional[pd.DataFrame] 261 An optional DataFrame or equivalent to sync into the pipe. 262 Defaults to `None`. 263 264 begin: Union[datetime, int, None], default None 265 Optionally specify the earliest datetime to search for data. 266 Defaults to `None`. 267 268 end: Union[datetime, int, None], default None 269 Optionally specify the latest datetime to search for data. 270 Defaults to `None`. 271 272 chunksize: Optional[int], default -1 273 Specify the number of rows to sync per chunk. 274 If `-1`, resort to system configuration (default is `900`). 275 A `chunksize` of `None` will sync all rows in one transaction. 276 Defaults to `-1`. 277 278 check_existing: bool, default True 279 If `True`, pull and diff with existing data from the pipe. Defaults to `True`. 280 281 debug: bool, default False 282 Verbosity toggle. Defaults to False. 283 284 Returns 285 ------- 286 A `SuccessTuple` of success (`bool`) and message (`str`). 287 """
Sync a pipe using a database connection.
Parameters
- pipe (mrsm.Pipe): The Meerschaum Pipe instance into which to sync the data.
- df (Optional[pd.DataFrame]):
An optional DataFrame or equivalent to sync into the pipe.
Defaults to
None. - begin (Union[datetime, int, None], default None):
Optionally specify the earliest datetime to search for data.
Defaults to
None. - end (Union[datetime, int, None], default None):
Optionally specify the latest datetime to search for data.
Defaults to
None. - chunksize (Optional[int], default -1):
Specify the number of rows to sync per chunk.
If
-1, resort to system configuration (default is900). AchunksizeofNonewill sync all rows in one transaction. Defaults to-1. - check_existing (bool, default True):
If
True, pull and diff with existing data from the pipe. Defaults toTrue. - debug (bool, default False): Verbosity toggle. Defaults to False.
Returns
- A
SuccessTupleof success (bool) and message (str).
289def create_pipe_indices( 290 self, 291 pipe: mrsm.Pipe, 292 debug: bool = False, 293 **kwargs: Any 294) -> mrsm.SuccessTuple: 295 """ 296 Create a pipe's indices. 297 298 Parameters 299 ---------- 300 pipe: mrsm.Pipe 301 The pipe whose indices need to be created. 302 303 Returns 304 ------- 305 A `SuccessTuple` indicating success. 306 """ 307 return False, f"Cannot create indices for instance connectors of type '{self.type}'."
Create a pipe's indices.
Parameters
- pipe (mrsm.Pipe): The pipe whose indices need to be created.
Returns
- A
SuccessTupleindicating success.
309def clear_pipe( 310 self, 311 pipe: mrsm.Pipe, 312 begin: Union[datetime, int, None] = None, 313 end: Union[datetime, int, None] = None, 314 params: Optional[Dict[str, Any]] = None, 315 debug: bool = False, 316 **kwargs: Any 317) -> mrsm.SuccessTuple: 318 """ 319 Delete rows within `begin`, `end`, and `params`. 320 321 Parameters 322 ---------- 323 pipe: mrsm.Pipe 324 The pipe whose rows to clear. 325 326 begin: datetime | int | None, default None 327 If provided, remove rows >= `begin`. 328 329 end: datetime | int | None, default None 330 If provided, remove rows < `end`. 331 332 params: dict[str, Any] | None, default None 333 If provided, only remove rows which match the `params` filter. 334 335 Returns 336 ------- 337 A `SuccessTuple` indicating success. 338 """ 339 raise NotImplementedError
Delete rows within begin, end, and params.
Parameters
- pipe (mrsm.Pipe): The pipe whose rows to clear.
- begin (datetime | int | None, default None):
If provided, remove rows >=
begin. - end (datetime | int | None, default None):
If provided, remove rows <
end. - params (dict[str, Any] | None, default None):
If provided, only remove rows which match the
paramsfilter.
Returns
- A
SuccessTupleindicating success.
341def get_pipe_data( 342 self, 343 pipe: mrsm.Pipe, 344 select_columns: Optional[List[str]] = None, 345 omit_columns: Optional[List[str]] = None, 346 begin: Union[datetime, int, None] = None, 347 end: Union[datetime, int, None] = None, 348 params: Optional[Dict[str, Any]] = None, 349 debug: bool = False, 350 **kwargs: Any 351) -> Union['pd.DataFrame', None]: 352 """ 353 Query a pipe's target table and return the DataFrame. 354 355 Parameters 356 ---------- 357 pipe: mrsm.Pipe 358 The pipe with the target table from which to read. 359 360 select_columns: list[str] | None, default None 361 If provided, only select these given columns. 362 Otherwise select all available columns (i.e. `SELECT *`). 363 364 omit_columns: list[str] | None, default None 365 If provided, remove these columns from the selection. 366 367 begin: datetime | int | None, default None 368 The earliest `datetime` value to search from (inclusive). 369 370 end: datetime | int | None, default None 371 The lastest `datetime` value to search from (exclusive). 372 373 params: dict[str | str] | None, default None 374 Additional filters to apply to the query. 375 376 Returns 377 ------- 378 The target table's data as a DataFrame. 379 """ 380 if type(self).get_pipe_docs is get_pipe_docs: 381 raise NotImplementedError( 382 f"Missing `get_pipe_data()` or `get_pipe_docs()` for {type(self)}." 383 ) 384 385 docs = self.get_pipe_docs( 386 pipe=pipe, 387 select_columns=select_columns, 388 omit_columns=omit_columns, 389 begin=begin, 390 end=end, 391 params=params, 392 debug=debug, 393 **kwargs 394 ) 395 if not docs: 396 return None 397 398 pd = mrsm.attempt_import('pandas') 399 try: 400 return pd.DataFrame(docs) 401 except Exception as e: 402 from meerschaum.utils.warnings import warn 403 warn(f"Cannot build DataFrame from pipe docs:\n{e}") 404 405 return None
Query a pipe's target table and return the DataFrame.
Parameters
- pipe (mrsm.Pipe): The pipe with the target table from which to read.
- select_columns (list[str] | None, default None):
If provided, only select these given columns.
Otherwise select all available columns (i.e.
SELECT *). - omit_columns (list[str] | None, default None): If provided, remove these columns from the selection.
- begin (datetime | int | None, default None):
The earliest
datetimevalue to search from (inclusive). - end (datetime | int | None, default None):
The lastest
datetimevalue to search from (exclusive). - params (dict[str | str] | None, default None): Additional filters to apply to the query.
Returns
- The target table's data as a DataFrame.
407def get_pipe_docs( 408 self, 409 pipe: mrsm.Pipe, 410 select_columns: Optional[List[str]] = None, 411 omit_columns: Optional[List[str]] = None, 412 begin: Union[datetime, int, None] = None, 413 end: Union[datetime, int, None] = None, 414 params: Optional[Dict[str, Any]] = None, 415 debug: bool = False, 416 **kwargs: Any 417) -> list[dict[str, Any]]: 418 """ 419 Return a pipe's data as a list of documents. 420 Defaults to `get_pipe_data().to_dict(orient='records')`. 421 422 Parameters 423 ---------- 424 pipe: mrsm.Pipe 425 The pipe with the target table from which to read. 426 427 select_columns: list[str] | None, default None 428 If provided, only select these given columns. 429 Otherwise select all available columns (i.e. `SELECT *`). 430 431 omit_columns: list[str] | None, default None 432 If provided, remove these columns from the selection. 433 434 begin: datetime | int | None, default None 435 The earliest `datetime` value to search from (inclusive). 436 437 end: datetime | int | None, default None 438 The lastest `datetime` value to search from (exclusive). 439 440 params: dict[str | str] | None, default None 441 Additional filters to apply to the query. 442 443 Returns 444 ------- 445 The target table's data as a list of dictionaries. 446 """ 447 df = self.get_pipe_data( 448 pipe=pipe, 449 select_columns=select_columns, 450 omit_columns=omit_columns, 451 begin=begin, 452 end=end, 453 params=params, 454 debug=debug, 455 **kwargs 456 ) 457 if df is None or df.empty: 458 return [] 459 return df.to_dict(orient='records')
Return a pipe's data as a list of documents.
Defaults to get_pipe_data().to_dict(orient='records').
Parameters
- pipe (mrsm.Pipe): The pipe with the target table from which to read.
- select_columns (list[str] | None, default None):
If provided, only select these given columns.
Otherwise select all available columns (i.e.
SELECT *). - omit_columns (list[str] | None, default None): If provided, remove these columns from the selection.
- begin (datetime | int | None, default None):
The earliest
datetimevalue to search from (inclusive). - end (datetime | int | None, default None):
The lastest
datetimevalue to search from (exclusive). - params (dict[str | str] | None, default None): Additional filters to apply to the query.
Returns
- The target table's data as a list of dictionaries.
461@abc.abstractmethod 462def get_sync_time( 463 self, 464 pipe: mrsm.Pipe, 465 params: Optional[Dict[str, Any]] = None, 466 newest: bool = True, 467 debug: bool = False, 468 **kwargs: Any 469) -> datetime | int | None: 470 """ 471 Return the most recent value for the `datetime` axis. 472 473 Parameters 474 ---------- 475 pipe: mrsm.Pipe 476 The pipe whose collection contains documents. 477 478 params: dict[str, Any] | None, default None 479 Filter certain parameters when determining the sync time. 480 481 newest: bool, default True 482 If `True`, return the maximum value for the column. 483 484 Returns 485 ------- 486 The largest `datetime` or `int` value of the `datetime` axis. 487 """
Return the most recent value for the datetime axis.
Parameters
- pipe (mrsm.Pipe): The pipe whose collection contains documents.
- params (dict[str, Any] | None, default None): Filter certain parameters when determining the sync time.
- newest (bool, default True):
If
True, return the maximum value for the column.
Returns
- The largest
datetimeorintvalue of thedatetimeaxis.
489@abc.abstractmethod 490def get_pipe_columns_types( 491 self, 492 pipe: mrsm.Pipe, 493 debug: bool = False, 494 **kwargs: Any 495) -> Dict[str, str]: 496 """ 497 Return the data types for the columns in the target table for data type enforcement. 498 499 Parameters 500 ---------- 501 pipe: mrsm.Pipe 502 The pipe whose target table contains columns and data types. 503 504 Returns 505 ------- 506 A dictionary mapping columns to data types. 507 """
Return the data types for the columns in the target table for data type enforcement.
Parameters
- pipe (mrsm.Pipe): The pipe whose target table contains columns and data types.
Returns
- A dictionary mapping columns to data types.
509def get_pipe_columns_indices( 510 self, 511 debug: bool = False, 512) -> Dict[str, List[Dict[str, str]]]: 513 """ 514 Return a dictionary mapping columns to metadata about related indices. 515 516 Parameters 517 ---------- 518 pipe: mrsm.Pipe 519 The pipe whose target table has related indices. 520 521 Returns 522 ------- 523 A list of dictionaries with the keys "type" and "name". 524 525 Examples 526 -------- 527 >>> pipe = mrsm.Pipe('demo', 'shirts', columns={'primary': 'id'}, indices={'size_color': ['color', 'size']}) 528 >>> pipe.sync([{'color': 'red', 'size': 'M'}]) 529 >>> pipe.get_columns_indices() 530 {'id': [{'name': 'demo_shirts_pkey', 'type': 'PRIMARY KEY'}], 'color': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}], 'size': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}]} 531 """ 532 return {}
Return a dictionary mapping columns to metadata about related indices.
Parameters
- pipe (mrsm.Pipe): The pipe whose target table has related indices.
Returns
- A list of dictionaries with the keys "type" and "name".
Examples
>>> pipe = mrsm.Pipe('demo', 'shirts', columns={'primary': 'id'}, indices={'size_color': ['color', 'size']})
>>> pipe.sync([{'color': 'red', 'size': 'M'}])
>>> pipe.get_columns_indices()
{'id': [{'name': 'demo_shirts_pkey', 'type': 'PRIMARY KEY'}], 'color': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}], 'size': [{'name': 'IX_demo_shirts_color_size', 'type': 'INDEX'}]}
534def get_pipe_size( 535 self, 536 pipe: mrsm.Pipe, 537 debug: bool = False, 538 **kwargs: Any 539) -> Union[int, None]: 540 """ 541 Return the on-disk size of a pipe's target table in bytes. 542 543 Parameters 544 ---------- 545 pipe: mrsm.Pipe 546 The pipe whose target table size to measure. 547 548 Returns 549 ------- 550 An `int` of the number of bytes occupied by the target table, 551 or `None` if the size cannot be determined. 552 """ 553 raise NotImplementedError( 554 f"`get_pipe_size()` is not implemented for instance connectors of type '{self.type}'." 555 )
Return the on-disk size of a pipe's target table in bytes.
Parameters
- pipe (mrsm.Pipe): The pipe whose target table size to measure.
Returns
- An
intof the number of bytes occupied by the target table, - or
Noneif the size cannot be determined.
557def compress_pipe( 558 self, 559 pipe: mrsm.Pipe, 560 debug: bool = False, 561 **kwargs: Any 562) -> mrsm.SuccessTuple: 563 """ 564 Compress a pipe's target table to reduce disk usage. 565 566 Parameters 567 ---------- 568 pipe: mrsm.Pipe 569 The pipe whose target table to compress. 570 571 Returns 572 ------- 573 A `SuccessTuple` indicating success. 574 """ 575 return False, ( 576 f"Compression is not supported for instance connectors of type '{self.type}'." 577 )
Compress a pipe's target table to reduce disk usage.
Parameters
- pipe (mrsm.Pipe): The pipe whose target table to compress.
Returns
- A
SuccessTupleindicating success.
579def decompress_pipe( 580 self, 581 pipe: mrsm.Pipe, 582 debug: bool = False, 583 **kwargs: Any 584) -> mrsm.SuccessTuple: 585 """ 586 Decompress a pipe's target table, the inverse of `compress_pipe()`. 587 588 Parameters 589 ---------- 590 pipe: mrsm.Pipe 591 The pipe whose target table to decompress. 592 593 Returns 594 ------- 595 A `SuccessTuple` indicating success. 596 """ 597 return False, ( 598 f"Decompression is not supported for instance connectors of type '{self.type}'." 599 )
Decompress a pipe's target table, the inverse of compress_pipe().
Parameters
- pipe (mrsm.Pipe): The pipe whose target table to decompress.
Returns
- A
SuccessTupleindicating success.
601def vacuum_pipe( 602 self, 603 pipe: mrsm.Pipe, 604 debug: bool = False, 605 **kwargs: Any 606) -> mrsm.SuccessTuple: 607 """ 608 Reclaim disk space from a pipe's target table. 609 610 Parameters 611 ---------- 612 pipe: mrsm.Pipe 613 The pipe whose target table to vacuum. 614 615 Returns 616 ------- 617 A `SuccessTuple` indicating success. 618 """ 619 return False, ( 620 f"Vacuuming is not supported for instance connectors of type '{self.type}'." 621 )
Reclaim disk space from a pipe's target table.
Parameters
- pipe (mrsm.Pipe): The pipe whose target table to vacuum.
Returns
- A
SuccessTupleindicating success.
623def analyze_pipe( 624 self, 625 pipe: mrsm.Pipe, 626 debug: bool = False, 627 **kwargs: Any 628) -> mrsm.SuccessTuple: 629 """ 630 Refresh the planner statistics for a pipe's target table. 631 632 Parameters 633 ---------- 634 pipe: mrsm.Pipe 635 The pipe whose target table to analyze. 636 637 Returns 638 ------- 639 A `SuccessTuple` indicating success. 640 """ 641 return False, ( 642 f"Analyzing is not supported for instance connectors of type '{self.type}'." 643 )
Refresh the planner statistics for a pipe's target table.
Parameters
- pipe (mrsm.Pipe): The pipe whose target table to analyze.
Returns
- A
SuccessTupleindicating success.
645def partition_pipe( 646 self, 647 pipe: mrsm.Pipe, 648 chunk_minutes: Optional[int] = None, 649 debug: bool = False, 650 **kwargs: Any 651) -> mrsm.SuccessTuple: 652 """ 653 Rebuild a pipe's target table to a new partition (chunk) width. 654 655 Parameters 656 ---------- 657 pipe: mrsm.Pipe 658 The partitioned pipe whose target table to repartition. 659 660 chunk_minutes: Optional[int], default None 661 The new partition width in minutes. Defaults to the pipe's `verify.chunk_minutes`. 662 663 Returns 664 ------- 665 A `SuccessTuple` indicating success. 666 """ 667 return False, ( 668 f"Repartitioning is not supported for instance connectors of type '{self.type}'." 669 )
Rebuild a pipe's target table to a new partition (chunk) width.
Parameters
- pipe (mrsm.Pipe): The partitioned pipe whose target table to repartition.
- chunk_minutes (Optional[int], default None):
The new partition width in minutes. Defaults to the pipe's
verify.chunk_minutes.
Returns
- A
SuccessTupleindicating success.
279def make_connector(cls, _is_executor: bool = False): 280 """ 281 Register a class as a `Connector`. 282 The `type` will be the lower case of the class name, without the suffix `connector`. 283 284 Parameters 285 ---------- 286 instance: bool, default False 287 If `True`, make this connector type an instance connector. 288 This requires implementing the various pipes functions and lots of testing. 289 290 Examples 291 -------- 292 >>> import meerschaum as mrsm 293 >>> from meerschaum.connectors import make_connector, Connector 294 >>> 295 >>> @make_connector 296 >>> class FooConnector(Connector): 297 ... REQUIRED_ATTRIBUTES: list[str] = ['username', 'password'] 298 ... 299 >>> conn = mrsm.get_connector('foo:bar', username='dog', password='cat') 300 >>> print(conn.username, conn.password) 301 dog cat 302 >>> 303 """ 304 import re 305 from meerschaum.plugins import _get_parent_plugin 306 suffix_regex = ( 307 r'connector$' 308 if not _is_executor 309 else r'executor$' 310 ) 311 plugin_name = _get_parent_plugin(2) 312 typ = re.sub(suffix_regex, '', cls.__name__.lower()) 313 with _locks['types']: 314 types[typ] = cls 315 with _locks['custom_types']: 316 custom_types.add(typ) 317 if plugin_name: 318 with _locks['plugins_types']: 319 if plugin_name not in plugins_types: 320 plugins_types[plugin_name] = [] 321 plugins_types[plugin_name].append(typ) 322 with _locks['connectors']: 323 if typ not in connectors: 324 connectors[typ] = {} 325 if getattr(cls, 'IS_INSTANCE', False): 326 with _locks['instance_types']: 327 if typ not in instance_types: 328 instance_types.append(typ) 329 330 return cls
Register a class as a Connector.
The type will be the lower case of the class name, without the suffix connector.
Parameters
- instance (bool, default False):
If
True, make this connector type an instance connector. This requires implementing the various pipes functions and lots of testing.
Examples
>>> import meerschaum as mrsm
>>> from meerschaum.connectors import make_connector, Connector
>>>
>>> @make_connector
>>> class FooConnector(Connector):
... REQUIRED_ATTRIBUTES: list[str] = ['username', 'password']
...
>>> conn = mrsm.get_connector('foo:bar', username='dog', password='cat')
>>> print(conn.username, conn.password)
dog cat
>>>
53def entry( 54 sysargs: Union[List[str], str, None] = None, 55 _patch_args: Optional[Dict[str, Any]] = None, 56 _use_cli_daemon: bool = True, 57 _session_id: Optional[str] = None, 58) -> SuccessTuple: 59 """ 60 Parse arguments and launch a Meerschaum action. 61 62 Returns 63 ------- 64 A `SuccessTuple` indicating success. 65 """ 66 start = time.perf_counter() 67 from meerschaum.config.environment import get_daemon_env_vars 68 sysargs_list = shlex.split(sysargs) if isinstance(sysargs, str) else sysargs 69 if ( 70 not _use_cli_daemon 71 or (not sysargs or (sysargs[0] and sysargs[0].startswith('-'))) 72 or '--no-daemon' in sysargs_list 73 or '--daemon' in sysargs_list 74 or '-d' in sysargs_list 75 or get_daemon_env_vars() 76 or not mrsm.get_config('system', 'experimental', 'cli_daemon') 77 ): 78 success, msg = entry_without_daemon(sysargs, _patch_args=_patch_args) 79 end = time.perf_counter() 80 if '--debug' in sysargs_list: 81 print(f"Duration without daemon: {round(end - start, 3)}") 82 return success, msg 83 84 from meerschaum._internal.cli.entry import entry_with_daemon 85 success, msg = entry_with_daemon(sysargs, _patch_args=_patch_args) 86 end = time.perf_counter() 87 if '--debug' in sysargs_list: 88 print(f"Duration with daemon: {round(end - start, 3)}") 89 return success, msg