Cross Validation

As time-series have the inherent structure we could run into problems with traditional shuffled Kfolds cross-validation. hcrystalball implements forward rolling cross-validation making training set consist only of observations that occurred prior to the observations that form the test set.

0d4c653808f5440393ac0c4bf6356136

[1]:

from hcrystalball.model_selection import FinerTimeSplit
from sklearn.model_selection import cross_validate
from hcrystalball.wrappers import ExponentialSmoothingWrapper

[2]:

import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('seaborn')
plt.rcParams['figure.figsize'] = [12, 6]

[3]:

from hcrystalball.utils import get_sales_data

df = get_sales_data(n_dates=100,
                    n_assortments=1,
                    n_states=1,
                    n_stores=1)
X, y = pd.DataFrame(index=df.index), df['Sales']

Native Cross Validation

[4]:

cross_validate(ExponentialSmoothingWrapper(),
               X,
               y,
               cv=FinerTimeSplit(horizon=5, n_splits=2),
               scoring='neg_mean_absolute_error')

[4]:

{'fit_time': array([0.00605536, 0.00434923]),
 'score_time': array([0.00402021, 0.00340152]),
 'test_score': array([-4829.36876279, -5350.33892   ])}

Grid search and model selection

Model selection and parameter tuning is the area where hcrystalball really shines. There is ongoing and probably a never-ending discussion about superiority or inferiority of ML techniques over common statistical/econometrical ones. Why not try both? The problem of a simple comparison between the performance of different kind of algorithms such as SARIMAX, Prophet, regularized linear models, and XGBoost lead to hcrystalball. Let’s see how to do it!

[5]:

from hcrystalball.compose import TSColumnTransformer
from hcrystalball.feature_extraction import SeasonalityTransformer
from hcrystalball.wrappers import ProphetWrapper
from hcrystalball.wrappers import get_sklearn_wrapper
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline

from hcrystalball.wrappers import SarimaxWrapper
from sklearn.model_selection import GridSearchCV

import numpy as np
import pandas as pd

Define our pipeline

[6]:

sklearn_model_pipeline = Pipeline([
    ('seasonality', SeasonalityTransformer(freq='D')),
    ('model', 'passthrough') # this will be overwritten by param grid
])

Define pipeline parameters including different models

[7]:

param_grid = [{'model': [sklearn_model_pipeline],
               'model__model':[get_sklearn_wrapper(RandomForestRegressor, random_state=42),
                               get_sklearn_wrapper(LinearRegression)]},
              {'model': [ProphetWrapper()],
               'model__seasonality_mode':['multiplicative', 'additive']},
              {'model': [SarimaxWrapper(order=(2,1,1), suppress_warnings=True)]}
             ]

Custom scorer

[11]:

from hcrystalball.metrics import make_ts_scorer
from sklearn.metrics import mean_absolute_error

[12]:

scoring = make_ts_scorer(mean_absolute_error,
                         greater_is_better=False)

[13]:

grid_search = GridSearchCV(estimator=sklearn_model_pipeline,
                           param_grid=param_grid,
                           scoring=scoring,
                           cv=FinerTimeSplit(horizon=5, n_splits=2),
                           refit=False,
                           error_score=np.nan)
results = grid_search.fit(X, y)

[14]:

results.scorer_.cv_data.loc[:,lambda x: x.columns != 'split'].plot();

[14]:

<AxesSubplot:>
../../../_images/examples_tutorial_wrappers_05_model_selection_22_1.png

hcrystalball internally tracks data based on unique model hashes since model string represantations (reprs) are very long for usable columns names in dataframe, but if you are curious i.e. what was the worse model not to use it for further experiment, you can do it with scorers estimator_ids attribute

[15]:

results.scorer_.cv_data.head()

[15]:
split y_true b1498790399b998a7f5c77fc18d9747e e36df968187a3c0c44635aca7e0dc85e e4b5e16b199ee974f179721c8f1a919d 7ffb3d592d08a22eb7ca50d0a5bc7de1 0849a3d8a8efd917f8e35849aec3384c
2015-07-22 0 18046.0 21179.66 20676.741135 24428.095232 24065.094696 18366.863642
2015-07-23 0 19532.0 21056.79 18015.494274 21852.187052 21566.265711 18798.317058
2015-07-24 0 17420.0 22834.67 18084.973453 20794.186776 20548.954160 19041.127406
2015-07-25 0 13558.0 13248.01 12739.066714 14475.237140 14713.923592 19029.719927
2015-07-26 0 0.0 243.32 -685.019580 -12.321919 1099.681532 19024.950811

We can get to the model definitions using hash in results.scorer_.estimator_ids dict