Seasonalities and Holidays

Despite that time-series modeling is not super feature-rich, we can still leverage at least deterministic features which are known in advance - calendar patterns.

The easiest way how to model seasonality and leverage the power of ML models is to create a binary feature for each repeating calendar pattern (day of the week, the month of the year…). This is implemented in the SeasonalityTransformer.

[1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')
plt.rcParams['figure.figsize'] = [12, 6]
[2]:
from hcrystalball.utils import get_sales_data

df = get_sales_data(n_dates=365*2,
                    n_assortments=1,
                    n_states=1,
                    n_stores=1)[:"2014-12-31"]
X, y = pd.DataFrame(index=df.index), df['Sales']

Seasonality Features

[3]:
from hcrystalball.feature_extraction import SeasonalityTransformer
SeasonalityTransformer(freq = 'D').fit(X, y).transform(X).head()
[3]:
_Friday _Monday _Saturday _Sunday _Thursday _Tuesday _Wednesday _1_week _2_week _3_week ... _May _November _October _September _1_quarter _2_quarter _3_quarter _4_quarter _2013 _2014
Date
2013-08-01 0 0 0 0 1 0 0 0 0 0 ... 0 0 0 0 0 0 1 0 1 0
2013-08-02 1 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 1 0 1 0
2013-08-03 0 0 1 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 1 0 1 0
2013-08-04 0 0 0 1 0 0 0 0 0 0 ... 0 0 0 0 0 0 1 0 1 0
2013-08-05 0 1 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 1 0 1 0

5 rows × 77 columns

Holiday Features

One of the important parts of time-series modeling is taking into account holidays which in most cases exhibit different behavior than it’s common. hcrystalball implements HolidayTransformer which returns column with string name of the holiday based on provided country ISO code for each date in the dataset (empty string if it’s not a holiday). All hcrystalball wrappers accept the output of HolidayTransformer and transform it into individual model formats. HolidayTransformer also supports some region specific holidays i.e. Germany state specific holidays, in that case the provided string shoule be in form country-region: i.e. ‘DE-HE’.

[4]:
from hcrystalball.feature_extraction import HolidayTransformer
HolidayTransformer(country_code = 'DE').fit(X, y).transform(X).tail(10)
[4]:
_holiday_DE
Date
2014-12-22
2014-12-23
2014-12-24
2014-12-25 Christmas Day
2014-12-26 Second Christmas Day
2014-12-27
2014-12-28
2014-12-29
2014-12-30
2014-12-31

The holiday effects have very often impact on target variable even before or after holidays (i.e. whole Christmas period is affected by few public holidays), in order to model such effects around holiday HolidayTransformer provides 3 parameters - days_before - number of days the before public holiday which should be taken into account, days_after - number of days before the public holiday which should be taken into account and bool variable bridge_days which will create variable for the overlaping days_before and days_after feature - mostly for modeling working days in the middle of two public holidays.

[5]:
from hcrystalball.feature_extraction import HolidayTransformer
HolidayTransformer(country_code = 'DE', days_before = 2, days_after = 1, bridge_days = True).fit(X, y).transform(X)['2014-04-16':'2014-04-23']
[5]:
_holiday_DE _2_before_holiday_DE _1_after_holiday_DE _bridge_holiday_DE
Date
2014-04-16 True False False
2014-04-17 True False False
2014-04-18 Good Friday False False False
2014-04-19 True True True
2014-04-20 True False False
2014-04-21 Easter Monday False False False
2014-04-22 False True False
2014-04-23 False False False

In Pipelines

It is possible to combine more countries/regions for the holidays. The behovior of multi holidays is wrapper specific.

[6]:
from hcrystalball.compose import TSColumnTransformer
from hcrystalball.wrappers import ExponentialSmoothingWrapper
from hcrystalball.wrappers import get_sklearn_wrapper
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
[7]:
from hcrystalball.wrappers import SarimaxWrapper
[8]:
pipeline = Pipeline([
    ('holidays_de', HolidayTransformer(country_code = 'DE')),
    ('holidays_be', HolidayTransformer(country_code = 'BE')),
    ('seasonality', SeasonalityTransformer(freq='D')),
    ('model', get_sklearn_wrapper(RandomForestRegressor, random_state=42))
])
[9]:
preds = (pipeline.fit(X[:-10], y[:-10])
         .predict(X[-10:])
         .merge(y, left_index=True, right_index=True, how='outer')
         .tail(50))

preds.plot(title=f"MAE:{(preds['Sales']-preds['sklearn']).abs().mean().round(3)}");
[9]:
<AxesSubplot:title={'center':'MAE:3838.153'}, xlabel='Date'>
../../../_images/examples_tutorial_wrappers_04_seasonalities_and_holidays_15_1.png

Also country code column can be used.

Note: This column is deleted in ``HolidayTransformer.transform`` method in order not to pollute the further processing

[10]:
pipeline_col = Pipeline([
    ('holidays_de', HolidayTransformer(country_code_column = 'germany')),
    ('holidays_be', HolidayTransformer(country_code_column = 'belgium')),
    ('seasonality', SeasonalityTransformer(freq='D')),
    ('model', get_sklearn_wrapper(RandomForestRegressor, random_state=42))
])
[11]:
X_col = X.copy().assign(germany='DE').assign(belgium='BE')
[12]:
preds_col = (pipeline_col.fit(X_col[:-10], y[:-10])
         .predict(X_col[-10:])
         .merge(y, left_index=True, right_index=True, how='outer')
         .tail(50))

preds_col.plot(title=f"MAE:{(preds_col['Sales'] - preds_col['sklearn']).abs().mean().round(3)}");
[12]:
<AxesSubplot:title={'center':'MAE:3838.153'}, xlabel='Date'>
../../../_images/examples_tutorial_wrappers_04_seasonalities_and_holidays_19_1.png

With exogenous variables

[13]:
X_col['trend'] = np.arange(len(X))
[14]:
preds_col = (pipeline_col.fit(X_col[:-10], y[:-10])
         .predict(X_col[-10:])
         .merge(y, left_index=True, right_index=True, how='outer')
         .tail(50))

preds_col.plot(title=f"MAE:{(preds_col['Sales'] - preds_col['sklearn']).abs().mean().round(3)}");
[14]:
<AxesSubplot:title={'center':'MAE:3677.595'}, xlabel='Date'>
../../../_images/examples_tutorial_wrappers_04_seasonalities_and_holidays_22_1.png

Individual holidays as separate columns

[15]:
sklearns_native_transformers = TSColumnTransformer(
    transformers=[
        ('one_hot_encoder', OneHotEncoder(sparse=False, drop='first'), ['_holiday_CZ'])
    ])

pipeline_col = Pipeline([
   ('holidays', HolidayTransformer(country_code='CZ')),
    ('one_hot_encoder', sklearns_native_transformers)
])
[16]:
pipeline_col.fit_transform(X_col['2013-12-20':'2013-12-26'])
[16]:
x0_Christmas Day x0_Christmas Eve x0_St. Stephen's Day (The Second Christmas Day)
Date
2013-12-20 0.0 0.0 0.0
2013-12-21 0.0 0.0 0.0
2013-12-22 0.0 0.0 0.0
2013-12-23 0.0 0.0 0.0
2013-12-24 0.0 1.0 0.0
2013-12-25 1.0 0.0 0.0
2013-12-26 0.0 0.0 1.0