Callable Functions
Generated step-first catalogue of callable functions sourced from fabricops_kit.__all__.
Notebook template quickstart
Use these notebook templates for an end-to-end lifecycle implementation:
00_env_config.ipynb: shared runtime setup (Step 1, Step 2A, Step 2B).02_ex_agreement_topic.ipynb: exploration/profiling and AI-assisted advisory suggestions with human decisions (Step 3, Step 4, Step 5, Step 8, Step 9).03_pc_agreement_source_to_target.ipynb: run-all-safe pipeline enforcement and controlled delivery (Step 1, Step 2B, Step 3, Step 6A-6D, Step 7, Step 10).
Clean split:
- 00_env_config = shared setup.
- 02_ex = exploration, profiling, AI suggestions, human decisions.
- 03_pc = approved enforcement and output.
Step 8 and Step 9 AI functions belong in exploration notebooks. Pipeline notebooks should enforce approved rules and should not make AI decisions at runtime.
Modules
| Module |
|---|
| ai |
| config |
| docs_metadata |
| drift |
| fabric_io |
| governance |
| handover |
| lineage |
| metadata |
| profiling |
| quality |
| run_summary |
| runtime |
| technical_columns |
Step 1: Governance context
This step captures the governance context: approved usage, owner, and data agreement. The agreement may live outside Fabric, such as in SharePoint documents. Functions in this step mainly link notebooks back to that agreement so the technical work stays tied to the approved business context.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
assert_notebook_name_valid |
runtime | Essential | Raise :class:NotebookNamingError when a notebook name is invalid. |
— |
build_runtime_context |
runtime | Essential | Build a standard runtime context dictionary for Fabric notebooks. | — |
configure_fabric_ai_functions |
ai | Essential | Apply optional default Fabric AI Function configuration. | — |
generate_run_id |
runtime | Essential | Generate a notebook-safe run identifier. | — |
validate_notebook_name |
runtime | Essential | Validate notebook names against the framework workspace notebook model. | — |
Step 2A: Create shared runtime config
This step creates the shared config that other notebooks depend on, including environment paths, workspace targets, AI availability, and standard naming rules. The goal is to define the project setup once so exploration and pipeline notebooks do not repeat hidden manual configuration.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
check_fabric_ai_functions_available |
config | Essential | Check whether Fabric AI Functions are available in the current runtime. | — |
create_ai_prompt_config |
config | Essential | Create the AI prompt-template configuration used by FabricOps. | — |
create_framework_config |
config | Essential | Create the top-level framework configuration object. | — |
create_governance_config |
config | Essential | Create governance-default configuration. | — |
create_lineage_config |
config | Essential | Create lineage-default configuration. | — |
create_notebook_runtime_config |
config | Essential | Create notebook naming-policy configuration for runtime guards. | — |
create_path_config |
config | Essential | Create environment-to-target routing used by Fabric IO helpers. | — |
create_quality_config |
config | Essential | Create quality-default configuration. | — |
get_path |
config | Essential | Resolve a configured Fabric path for an environment and target. | — |
Housepath |
fabric_io | Essential | Fabric lakehouse or warehouse connection details. | — |
load_fabric_config |
fabric_io | Essential | Validate and return a framework config mapping. | — |
validate_framework_config |
config | Essential | Validate and normalize framework config input. | — |
Step 2B: Run notebook startup checks
This step runs the startup utility or smoke test at the beginning of every exploration and pipeline notebook. The goal is to confirm the notebook is running in the expected environment, follows naming rules, and has the required Fabric or AI capabilities before any data work begins.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
bootstrap_fabric_env |
config | Essential | Bootstrap 00_env_config environment readiness by resolving required targets and collecting runtime/AI check results. | _get_fabric_runtime_metadata (internal) |
run_config_smoke_tests |
config | Essential | Run 00_env_config smoke checks for Spark, runtime context, configured paths, notebook naming, and optional AI/IO imports. | _check_spark_session (internal), _get_fabric_runtime_metadata (internal) |
Step 3: Define source contract & ingestion pattern
This step defines the contract between the upstream source and this notebook. It captures what data is expected, including schema, data types, update frequency, update method, watermark column, and whether the source is append only, overwritten, or slowly changing. Functions in this step help the pipeline decide how to ingest, validate, snapshot, or incrementally process the source data.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
lakehouse_csv_read |
fabric_io | Essential | Read a CSV file from a Fabric lakehouse Files path. | _get_spark (internal) |
lakehouse_excel_read_as_spark |
fabric_io | Essential | Read an Excel file from a Fabric lakehouse Files path. | _get_spark (internal) |
lakehouse_parquet_read_as_spark |
fabric_io | Essential | Read a Parquet file from a Fabric lakehouse Files path. | _convert_single_parquet_ns_to_us (internal), _get_spark (internal) |
lakehouse_table_read |
fabric_io | Essential | Read a Delta table from a Fabric lakehouse. | _get_spark (internal) |
load_data_contract |
quality | Essential | Load and normalize a data product contract from file path or dictionary. | — |
warehouse_read |
fabric_io | Essential | Read a table from a Microsoft Fabric warehouse. | _get_spark (internal) |
Step 4: Ingest, profile & store source data
This step brings the source data into the framework, profiles it, and stores it for later use. Functions in this step focus on reading the data, capturing basic profiling results, and saving the raw or source-aligned version before business transformation begins.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
build_ai_quality_context |
profiling | Essential | Build deterministic AI-ready context from standard metadata profile rows. | — |
build_schema_drift_records |
metadata | Essential | Convert schema drift results into metadata records for audit trails. | — |
build_schema_snapshot_records |
metadata | Essential | Convert a schema snapshot into row-wise metadata records. | — |
check_partition_drift |
drift | Essential | Check partition-level drift using keys, partitions, and optional watermark baselines. | — |
check_profile_drift |
drift | Essential | Compare profile metrics against a baseline profile and drift thresholds. | — |
check_schema_drift |
drift | Essential | Compare a current dataframe schema against a baseline schema snapshot. | — |
generate_metadata_profile |
profiling | Essential | Generate standard metadata profile rows for a Spark/Fabric DataFrame. | — |
profile_dataframe |
profiling | Essential | Build a lightweight profile for pandas or Spark-like DataFrames. | — |
profile_dataframe_to_metadata |
profiling | Essential | Profile a Spark/Fabric DataFrame into metadata-compatible metadata rows. | — |
profile_metadata_to_records |
profiling | Essential | Convert Spark metadata profile rows into JSON-friendly dictionaries. | — |
summarize_drift_results |
drift | Essential | Summarize schema, partition, and profile drift outcomes into one decision. | — |
Step 5: Explore data & explain transformation logic
This step is where the analyst studies the profiled source data and explains why transformation is needed. There may not be many helper functions here today, but future functions could support standard EDA, AI assisted analysis, and documentation of business assumptions before the logic becomes part of the repeatable pipeline.
No public callable currently mapped to this step.
No public callable is currently exported for this step. Use notebook prompts for AI-assisted rule drafting.
Step 6A: Write transformation logic
This step contains the main transformation logic that converts source-aligned data into the target output. Functions here support reusable pipeline code so the same logic can run consistently during development, testing, and scheduled refresh.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
run_data_product |
quality | Essential | Run the starter kit workflow end-to-end for a data product outcome. | _effective_contract_dict (internal), _refresh_mode (internal), _runtime_validation_contract (internal), _write_dataframe_to_table (internal), _write_records_spark (internal) |
Step 6B: Apply runtime standards
This step applies standard runtime requirements such as technical columns, run IDs, timestamps, partition keys, and other repeatable conventions. Functions here make outputs easier to audit, troubleshoot, join back to pipeline runs, and operate at scale.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
add_audit_columns |
technical_columns | Essential | Add run tracking and audit columns for ingestion workflows. | _assert_columns_exist (internal), _bucket_values_pandas (internal), _get_fabric_runtime_context (internal), _resolve_engine (internal) |
add_datetime_features |
technical_columns | Essential | Add localized datetime feature columns derived from a UTC datetime column. | _assert_columns_exist (internal), _resolve_engine (internal) |
add_hash_columns |
technical_columns | Essential | Add business key and row-level SHA256 hash columns. | _assert_columns_exist (internal), _hash_row (internal), _non_technical_columns (internal), _resolve_engine (internal) |
default_technical_columns |
technical_columns | Essential | Return framework-generated and legacy technical column names to ignore. | — |
Step 6C: Enforce pipeline controls
This step enforces the controls that decide whether the pipeline output should be trusted. Functions here support data quality rules, schema checks, classification checks, and other contract validations before data is released downstream.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
run_quality_rules |
quality | Essential | Execute quality rules against a dataframe and return structured results. | _normalize_severity (internal), _now_iso (internal), _pandas_rule (internal), _resolve_engine (internal), _result_from_counts (internal), _spark_rule (internal), _to_jsonable (internal) |
Step 6D: Write controlled outputs
This step writes the transformed output to the correct lakehouse, warehouse, or product layer. Functions here make the write pattern explicit, repeatable, and aligned to the intended environment instead of relying on ad hoc exports.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
lakehouse_table_write |
fabric_io | Essential | Write a Spark DataFrame to a Fabric lakehouse Delta table. | — |
warehouse_write |
fabric_io | Essential | Write a Spark DataFrame to a Microsoft Fabric warehouse table. | — |
Step 7: Profile output & publish product contract
This step profiles the created output, stores its metadata, and creates the data contract for the next notebook, pipeline, or consumer. Functions here help record what was produced, write the evidence to metadata tables or catalogues, and make the output understandable and reusable downstream.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
build_dataset_run_record |
metadata | Essential | Build a dataset-run metadata record for operational tracking. | — |
build_quality_result_records |
metadata | Essential | Convert quality-rule execution output into metadata evidence records. | — |
write_metadata_records |
metadata | Essential | Write metadata records to a configured metadata sink. | — |
write_multiple_metadata_outputs |
metadata | Essential | Write multiple metadata payloads to their configured destinations. | — |
Step 8: Suggest AI assisted data quality rules
This step uses AI in exploration notebooks to suggest possible data quality rules from profiling results, business context, and source knowledge. These AI functions are only advisory. The actual rule creation, approval, and enforcement must still be done by the human engineer in the pipeline notebook.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
build_dq_rule_candidate_prompt |
ai | Optional | Build the DQ-candidate prompt used in AI-assisted quality drafting. | _resolve_prompt_template (internal) |
build_manual_dq_rule_prompt_package |
ai | Optional | Build copy/paste prompt package for manual DQ candidate generation. | _compact_sample_rows (internal) |
generate_dq_rule_candidates_with_fabric_ai |
ai | Optional | Append AI-suggested DQ rule candidates to a profiling DataFrame. | _require_fabric_ai_dataframe (internal) |
Step 9: Suggest AI assisted column classification
This step uses AI in exploration notebooks to suggest column classifications such as PII, sensitivity level, and governance labels for the planned output. These AI functions are only advisory. The actual label assignment must be approved by governance or data stewards and enforced by the human engineer in the pipeline notebook.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
build_governance_candidate_prompt |
ai | Optional | Build the governance-candidate prompt for AI-assisted classification drafts. | _resolve_prompt_template (internal) |
build_governance_classification_records |
governance | Optional | Build metadata-ready governance classification records from column suggestions. | — |
build_manual_governance_prompt_package |
ai | Optional | Build copy/paste prompt package for manual governance suggestion generation. | _compact_sample_rows (internal) |
classify_column |
governance | Optional | Classify one column using term matching, metadata cues, and business context. | _match_terms (internal), _phrase_in_text (internal) |
classify_columns |
governance | Optional | Classify multiple columns and return normalized governance suggestions. | _column_name (internal), _normalize_columns (internal) |
generate_governance_candidates_with_fabric_ai |
ai | Optional | Execute Fabric AI Functions to append governance suggestions to a DataFrame. | _require_fabric_ai_dataframe (internal) |
summarize_governance_classifications |
governance | Optional | Summarize governance classification outputs into review-friendly counts. | — |
write_governance_classifications |
governance | Optional | Persist governance classifications to a metadata destination. | _spark_create_governance_metadata_dataframe (internal) |
Step 10: Generate lineage & handover documentation
This step creates the final documentation needed for review, handover, and future maintenance. Functions here support lineage, transformation summaries, and handover notes so another analyst or engineer can understand what was built, why it was built, and how to operate it.
| Function / class | Module | Importance | Purpose | Related helpers |
|---|---|---|---|---|
build_handover_summary_prompt |
ai | Optional | Build the handover-summary prompt for AI-assisted run handoff drafting. | _resolve_prompt_template (internal) |
build_lineage_from_notebook_code |
lineage | Optional | Scan, optionally enrich, and validate lineage from notebook source code. | — |
build_lineage_handover_markdown |
lineage | Optional | Create a concise markdown handover summary from lineage execution results. | — |
build_lineage_record_from_steps |
lineage | Optional | Create metadata-ready lineage records from validated lineage steps. | — |
build_lineage_records |
lineage | Optional | Build compact lineage records for downstream metadata sinks. | — |
build_manual_handover_prompt_package |
ai | Optional | Build copy/paste prompt package for manual handover summary generation. | _compact_sample_rows (internal) |
build_run_summary |
run_summary | Optional | Build a handover-friendly summary for one data product run. | — |
enrich_lineage_steps_with_ai |
lineage | Optional | Optionally enrich deterministic lineage steps using an AI helper callable. | — |
fallback_copilot_lineage_prompt |
lineage | Optional | Build a fallback Copilot prompt for manual lineage enrichment. | — |
generate_handover_summary_with_fabric_ai |
ai | Optional | Execute Fabric AI Functions to append handover summary suggestions. | _require_fabric_ai_dataframe (internal) |
parse_manual_ai_json_response |
ai | Optional | Parse manual AI JSON output into Python objects. | — |
plot_lineage_steps |
lineage | Optional | Render lineage steps as a directed graph figure. | — |
render_run_summary_markdown |
run_summary | Optional | Render a run summary dictionary into Markdown for handover notes. | _status_of (internal) |
scan_notebook_cells |
lineage | Optional | Scan multiple notebook cells and append cell references to lineage steps. | — |
scan_notebook_lineage |
lineage | Optional | Extract deterministic lineage steps from notebook code using AST parsing. | _call_name (internal), _flatten_chain (internal), _name (internal), _resolve_write_target (internal), _step (internal) |
validate_lineage_steps |
lineage | Optional | Validate lineage step structure and flag records requiring human review. | — |
Other exported callables
These callables are exported by fabricops_kit.__all__ but are not currently mapped to a lifecycle step. They are listed here so the public reference remains complete.
No unmapped exported callables.
Use these .ipynb templates by importing/copying them into Fabric and renaming the actual notebook to the required lifecycle naming convention. Notebook name validation reads the running notebook name from Fabric runtime context when available.