Skip to content

EvidentlyAIModelChecker

This class is used for checking and calculating model performance and drift using the Evidently AI library. It extends the BaseModelChecker class and provides two main methods: check_model and calculate_model_drift.

Configuration

Required Configuration

The EvidentlyAI model checker requires the following configuration:

  • local_dir: Location of a local directory to output files generated by this component.
  • model_target: The model target in the dataset

Optional Configuration

The EvidentlyAI model checker has no optional configuration.

Default Configuration

The EvidentlyAI model checker uses the following optional configuration:

  • evidentlyai_model_report_name: The file name of the generated report. Defaults to EVIDENTLYAI_MODEL_REPORT.HTML.
  • evidentlyai_model_drift_report_name: The file name of the generated report. Defaults to EVIDENTLYAI_MODEL_DRIFT_REPORT.HTML.

Methods

check_model

This method runs various test presets using EvidentlyAI, including test/train drift. 

 def check_model(self, data_dict, model, *args, **kwargs)

Arguments:

  • data_dict (dict): a dictionary containing the input data for the model. It should include a training set (X_train, y_train) and a test set (X_test, y_test).
  • model (object): the trained model object.

Returns:

  • model_report (object): The model suite report.
  • file_path (str): The location of the saved HTML file.
  • checks_status (str): The status (PASS, ERROR or WARN) of model checks.

Example:

from lolpop.component import EvidentlyAIModelChecker

... #set up data and model

data_dict = data_splitt.split(data)
model = hyperparameter_tuner.run_experiment(data_dict, model_version) 

config = {
    #insert component configuration here
}

# Instantiate the EvidentlyAIModelChecker class
model_checker = EvidentlyAIModelChecker(conf=config)

# Check the model performance
model_report, file_path, checks_status = model_checker.check_model(data_dict, model)

# Print the results
print("Checks Status: ", checks_status)

calculate_model_drift

This method is used to calculate the drift between the predictions of a current model and a deployed model.

calculate_model_drift(self, data, current_model, deployed_model, *args, **kwargs)

Arguments:

  • data (dict): a dictionary containing the input data for the drift calculation. It should include a test set (X_test, y_test).
  • current_model (object): the current model object.
  • deployed_model (object): the deployed model object.

Returns

  • model_report (object): The train/test drift report object.
  • file_path (str): The location of the saved HTML file.

Example:

from lolpop.component import EvidentlyAIModelChecker
... #set up data and model

data_dict = data_splitt.split(data)
model = hyperparameter_tuner.run_experiment(data_dict, model_version) 

config = {
    #insert component configuration here
}

# Instantiate the EvidentlyAIModelChecker class
model_checker = EvidentlyAIModelChecker(conf=config)

# Calculate the model drift
drift_report, file_path = model_checker.calculate_model_drift(data, current_model, deployed_model)