On the way towards a low-carbon future, electricity networks are considered as enablers and one of the critical areas to be studied under the Strategic Energy Technologies Plan. The first European Electricity Grid Initiative –EEGI– Roadmap 2010-2018 was approved by the European Commission and the Member States alongside the creation of EEGI in June 2010. The EEGI Roadmap defines the research, development and demonstration challenges that both European transmission and distribution system operators should address in the next years with the aim to face the requirements linked to the evolution of power systems and to respond to different external factors. For this reason, smart-grid projects are receiving a lot of attention (Hübner & Prüggler, 2011; Giordano et al., 2011; Losa, De Nigris & Van, 2013). New perspectives emerge for energy management. Many smart meters and sensors are being deployed and they result in a new data deluge we will have to face. With the rollout of smart metering infrastructure at scale, demand-response programs may now be tailored based on users’ consumption and production patterns as mined from sensed data.

Energy efficiency and sustainability are important factors to address in the context of smart cities. In this sense, a necessary functionality is to reveal various preferences, behaviors, and characteristics of individual consumers and prosumers, considering the fine-grained energy consumption and production information from smart meters, respectively. Smart metering and nonintrusive load monitoring play a crucial role in fighting energy thefts and for optimizing the energy consumption of the home, building, city, and so forth (Fenza, Gallo & Loia, 2019; Ahmad et al., 2018). Besides, it is very important to reduce the mismatch between the actual and expected energy demand, which is often due to an anomalous operation of the equipment and control systems. In this context, the characterization of energy consumption patterns over time is of fundamental importance (Capozzoli et al., 2018).

To the best of our knowledge, all approaches are still in a research phase, especially when it comes to clustering methods consumption and production data to provide clusters for each consumer and prosumer profile (Hossain et al., 2011; Binh et al., 2010; Figueiredo et al., 2005; Mutanen et al., 2011; Lee, Haben, & Grindrod, 2014; Ardakanian et al., 2014; Lavin & Klabjan, 2014).

While some authors have been working on grouping consumers considering the similarity among time series models, such as ARMA and ARIMA (Brockwell & Davis, 2002); others have been focusing on grouping consumers considering the time series as feature vectors (Flath et al., 2012; Cao, Beckel, & Staake, 2013). Most of the proposals model the data following only a local point of view, others only global, and others considering the original time series, which limits the analysis. The most successful approaches have been those that combine various clustering methods (Figueiredo et al., 2005; Albert & Rajagopal, 2013; Räsänen et al., 2010; Räsänen & Kolehmainen, 2009). It is even possible to find some proposals that combine clustering with other machine learning techniques, such as association rules (Funde et al., 2019). Nevertheless, those hybrid approaches only exploit the combination of clustering methods in order to mitigate the disadvantages of ones and enhance the benefits of others. However, they do not exploit other important reasons for developing hybrid time series clustering models.

Due to the limitations expressed above, in this paper, we introduce a general methodology that combines clustering methods in two stages and exploits in a hybrid way local and global patterns of the series under analysis. Our proposal can be used to group energy consumers and prosumers according to the similarity of their daily and yearly consumption and production, respectively. Thus, characteristic load and production profiles per time period can be determined, as we will explain later. The obtained results are generally applicable and will be useful in a general business analytics context.

Smart meter data are time series; which makes the analysis quite complex. For that reason, cluster analysis of consumption data has been explored in some papers (Hossain et al., 2011; Binh et al., 2010; Figueiredo et al., 2005; Mutanen et al., 2011; Lee et al., 2014; Ardakanian et al., 2014; Lavin & Klabjan, 2014), not so much the clustering of production data. From now on we will refer to consumption data clustering approaches; however, all proposals are applicable to production data clustering as well. Most authors have been focused on grouping consumers considering the time series as feature vectors. In literature four approaches are proposed to cope with the feature vector definition:

Consider features as interval consumption measurements (e.g., every 15 minutes) (Flath et al., 2012; Cao et al., 2013).

The first one follows a raw-data-based approach, the last two follow a feature-based approach and the third one considers an extension of the raw data including other features.

The definition of a distance measure between time series is necessary for the four approaches (Iglesias & Kastner, 2013), and it depends on the clustering objective, which can be similarity in time, similarity in shape or similarity in change (Zhang et al., 2011):

The similarity in time is to cluster together series that vary in a similar way on each time level, as shown in Figure 1 . Usually, the clustering of time series data based on similarity in time requires the calculation of the exact distances among all the time series data in a dataset.

Figure 1 Figure 1 Time series based on similarity in time Source: Zhang et al. (2011).

Figure 2 Figure 2 Time series based on similarity in shape Source: Zhang et al. (2011).

Figure 3 Figure 3 Time series based on similarity in change Source: Zhang et al. (2011).

In this research, we are interested in time series clustering where the main clustering objective is the similarity in time because we need to cluster together series that vary in a similar way at each time interval. For this reason, in the first approach, it is necessary to define a distance measure based on the specific characteristics of time series data. Secondly, the arithmetic means of the single time segments are the starting point for the formation of global consumer behavior, but global features only do not properly represent the customers’ behavior. Thus, the second approach is not enough to segment the customers, and make groups of households with similar consumption patterns and determine on the fly the cluster membership of a given load curve. In the third approach, the dimensionality of the time series is increased and it could be difficult to manage different kinds of features, global and local in the same clustering process. Finally, the last approach could be useful for detecting clusters with similar load profiles, but it could be depending on the homogeneity of the data from the global feature point of view.

As we pointed out, the above approaches have some advantages and disadvantages. Thus, some authors prefer to develop hybrid approaches for time series clustering in other to solve the above disadvantages (Zhang et al., 2011; Lai et al., 2010; Aghabozorgi et al., 2014; Warren, 2007; Oates, Firoiu & Cohen, 1999; Aghabozorgi, Saybani & Wah, 2012). There are several reasons for developing hybrid time series clustering models (Lai et al., 2010). For instance, we might obtain very different clustering results for the same time series dataset when different time granules are considered. For time series clustering, dimensionality reduction methods are often applied to reduce the data dimension before clustering. Consequently, the information of subsequence may be overlooked. Therefore this might result in different clustering results after considering the subsequence information. Some conventional clustering methods require prior information and domain knowledge; others do not require prior information but are too computationally expensive to be applied on very large data sets. The combination of clustering methods can mitigate the disadvantages of some and enhance the benefits of others. For some applications, the clustering objective might not be that apparent. The selection of the time series representation and the similarity measure depends on the clustering objective. Thus, different clustering approaches are required.

Some hybrid clustering methods are proposed in the area of clustering analysis of smart metering data (Figueiredo et al., 2005; Albert & Rajagopal, 2013; Räsänen et al., 2010; Räsänen & Kolehmainen, 2009). Most of them apply Self-Organizing Maps –SOM– (Kohonen, 1982) in the first level and k-means (MacQueen, 1967) or hierarchical clustering algorithms (Räsänen et al., 2010) in the second level. SOM is used to obtain a reduction of the dimension of the initial dataset and k-means is used to group the weight vectors of the SOM’s units and the final clusters are obtained (Figueiredo et al., 2015; Räsänen et al., 2010; Räsänen & Kolehmainen, 2009). Another approach applies k-means first and uses spectral clustering to segment a collection into classes of similar statistical properties (Albert & Rajagopal, 2013). These hybrid approaches only exploit the combination of clustering methods in order to mitigate the disadvantages of ones and enhance the benefits of others. However, they do not exploit other important reasons for developing hybrid time series clustering models.

We introduce a general methodology that can be used to group time series considering different time granules. The proposed methodology consists of the following stages, as shown in Figure 4.

Figure 4 Figure 4 General schema of the proposed methodology Source: Own elaboration.

Stage 1: Data gathering. Read time series; e.g., the time intervals of interest for data representation are typical 1 min, 15 min or 1h in the context of smart metering applications (Chicco, 2012).

Stage 2: Data stratifying. Segment the data, which separates the raw data sets into more homogeneous subsets, in order to sustain scalability; e.g., data can be stratified using a split between weekend and weekdays, or between summer and winter months when we are working with smart metering data (Flath et al., 2012; Cao et al., 2013).

Stage 3: Data cleaning. Detect and remove errors and inconsistencies from data in order to improve the data quality. In the smart metering domain some strategies can discard data sets showing more than one hour of recording gaps (Flath et al., 2012); check for inconsistencies in the data and remove outliers (Figueiredo et al., 2005); detect missing values and replace them using regression techniques (Figueiredo et al., 2005); remove special days (e.g., public holidays) (Cao et al., 2013); remove non-continuous data (McLoughlin, Duffy & Conlon, 2012).

Stage 4: Different time granule clustering. Select the time granule before clustering. Depending on the granularity level desired in the clustering, it is defined all the elements that contribute to the clustering. This stage can be repeated several times depending on how much you want to refine the level of granularity in the data analysis. This is the most important stage of our methodology. For that reason, we will explain in detail its main steps:

4.1 Time granule selection

Stage 5: Post-clustering. Apply clustering validation techniques, visualize the clustering results and obtain labels and prototypes for each cluster. For example, a useful post-clustering result in the smart metering applications can be the calculation of the global power and energy information for the customer classes for tariff setting purposes (Chicco, 2012).

The selection of the level of granularity is closely associated with the objective of the desired clustering, as shown in Figure 5. In this step, it is necessary to decide if the objective is the similarity in shape, in change or in time. The selection of the time series representation (raw-data-based, feature-based or model-based representation) also depends on the clustering objective. For example, if the objective is the similarity in time then we suggest using raw-data-based representation. On the other hand, if the objective is the similarity in change, we suggest using a model-based representation. We can change the representation in different clustering levels.

Figure 5 Figure 5 Stage 4 schema Source: Own elaboration.

Then, preprocessing is in charge of applying dimensionality reduction and normalization methods in correspondence with the clustering objective and the selected representation. Finally, it is necessary to define the appropriate distance measure and apply a clustering algorithm for obtaining clusters where all the series grouped in the same cluster should be coherent or homogeneous. It is important to take into account which is our clustering objective for deciding which distance measure we will apply. For example, if the clustering objective is the similarity in shape, then it will be useful to apply Dynamic Time Warping –DTW– distance. The most used algorithms are k-means (Lavin & Klabjan, 2014; Flath et al., 2012; Räsänen & Kolehmainen, 2009), Self-Organizing Maps –SOM– (Figueiredo et al., 2005; McLoughlin et al., 2012), hierarchical approaches (Cao et al., 2013) and Expectation Maximization (Albert & Rajagopal, 2013); as well as hybrid approaches (Figueiredo et al., 2005; Albert & Rajagopal, 2013; Räsänen et al., 2010; Räsänen & Kolehmainen, 2009).

Stage 4 offers the possibility to design procedures for different clustering objectives using diverse granules in the time series representation. Taking into account the smart metering domain, it could be useful in the first clustering level to group consumption or production data considering their voltage level combined with general global features; thus, a normalization process, in the second level, could be carried out inclusive a similarity in time objective clustering. In the case of high dimensional series, it could be useful to apply a feature-based representation in the first clustering level, and after that, refine the clustering results considering the raw-representation in the second level.

In this section, we apply the general architecture outlined in the previous section, to the smart grid data introduced earlier. More specifically, we apply a two-level clustering approach. The first level of clustering is based on features extracted from the time series. Regardless of the length of the time series and missing values, a finite set of statistical measures is used to capture the nature of the time series. The feature values are obtained from each individual series and can be fed into some specific clustering algorithm. In the first level, we propose to cluster data using only global features in order to divide consumers or prosumers considering their daily consumption or production data, respectively. In the second level, we split the obtained clusters in the first level, considering local features for discovering sub-clusters for each consumption or production profile. Features are obtained by applying statistical operations that best capture the underlying characteristics of the time series, depending on the clustering objective. Thus, a feature-based representation is used at both levels; nevertheless, the clustering objective is different in each level. The main objective is clustering considering the similarity in time.

Transforming the raw time-series data into the set of features has been used by several authors (Räsänen & Kolehmainen, 2009; Wang, Smith & Hyndman, 2006; Wang et al., 2004; Nanopoulos, Alcock, & Manolopoulos, 2001). Feature-based representation has several advantages; we will mention some of them. When the time series is very long (high dimensionality), some clustering algorithms become intractable; for instance, fail because the similarity is dubious in high dimension space. Applying dimensionality reduction via feature extraction, we are able to cluster long length time series very efficiently. Despite the length of the time series and missing values, a finite set of statistical measures can be used to capture the global and local nature of the time series. Furthermore, feature extraction is used to compress large data sets by means of dimensionality reduction. When the clustering algorithm is based on a distance metric (e.g., Euclidean distance), it cannot handle time series with missing data or of different lengths if actual points are used as inputs. However, by extracting a set of measures from the original time series we simply bypass this problem. Computational efficiency can be increased and the use of more sophisticated clustering algorithms is possible. When the clustering objective is the similarity in shape, it is possible to obtain good results using a feature-based representation.

Nevertheless, feature-based representation has some disadvantages; we will mention some of them. The majority of feature extraction methods are generic in nature, the extracted features are usually application dependent. Thus one set of features that work well on one application might not be relevant to another. When the clustering objective is the similarity in time, it is not possible to obtain good results using global features extracted from the time series.

Considering the above-mentioned advantages and disadvantages, and bearing in mind the objective of the clustering at each level, we propose to obtain two kinds of features at each level. Figure 6 shows the general schema of the proposed two-level clustering approach based on local and global features.

Figure 6 Figure 6 General schema of our two-level clustering approach based on local and global features Source: Own elaboration.

In the first level, we propose to cluster data using only global features in order to divide consumptions or productions considering general behaviors. The principal global features to compute are mean, minimum, maximum and sum considering the total original features (e.g., 96 features if we consider 15 min interval consumption or production during a day). The clustering results can be improved if we include other features such as median, mode, standard deviation, variance, skewness, kurtosis, range, trend, seasonality, periodicity, serial correlation and chaos (Wang et al., 2006). Using a global feature-based representation, the dimensionality of the time series is significantly reduced and the clustering algorithm is much less sensitive to missing or noisy data. These features are enough to obtain clusters of consumers or prosumers with the same consumption or production levels and general characteristics, respectively; but they are not enough for generating clusters for each consumption or production profile per time period, because they cannot identify peaks at specific time periods.

In the second level, we split and refine the obtained clusters at the first level, considering local features for discovering sub-clusters. The local features express other information than the global features and emphasize the original time series characteristics. For characterizing the time series locally, it is possible to define a one hour, one week, or one day window, depending on the original time period. If we are processing daily profiles, a one hour window is used. A one day window is used for processing yearly profiles. We created four features for each window, computing mean, maximum, minimum and range, respectively. These local values allow expressing the time series behavior in each specified window. The clustering results can be improved if we include other features such as median, mode, standard deviation, variance, skewness, kurtosis, range, trend, seasonality, periodicity, serial correlation and chaos (Räsänen & Kolehmainen, 2009; Wang et al., 2006).

We currently explore suitable clustering methods and the choice of parameters that enable us to obtain general and specific clusters considering global and local features, respectively. The most used clustering methods are k-means (MacQueen, 1967), linkage (Defays, 1977), spectral clustering (Shi & Malik, 2000) and SOM (Kohonen, 1982). Useful similarity measures for these algorithms are Euclidean, cosine, correlation, and Manhattan (Iglesias & Kastner, 2013). We applied the k-means clustering algorithm (MacQueen, 1967) on global features in the first level for fixing a reasonable number of clusters to be obtained. The Complete Linkage clustering algorithm (Defays, 1977) was applied to the second level. The Euclidean distance was used for comparing the vectors in the cluster because it is a good distance when the clustering objective is the similarity in time.

In Belgium, the authority on energy policy is shared between the federal and the regional administrations. The competent authority for the smart metering roll-out in Flanders is the regional energy regulator, VREG, while there are two Distribution System Operators –DSO–: Eandis and Infrax (European Commission, 2014; Renner & Heinemann, 2011).

There are few studies on profile identification and consumer segmentation in Belgium (Espinoza et al., 2005; Alzate et al., 2009). The least recent result starts from consumption data containing hourly consumption values from substations within the Belgian grid. The typical daily profile for each consumer is first identified, and, after that, the k-means algorithm is applied for capturing the different profiles. A large dataset of over 1300 load profiles of residential customers forms the basis for modeling in the most recent result. Each load profile is a sequence of measured data, with a resolution of 15 min, over the duration of one year. A multiway spectral clustering without the use of pre-modeling steps was used to detect consumer profiles.

In the research presented in this paper, we use real-life data, provided by Eandis. Houses in Belgium are connected to the electricity grid of the DSO and they are organized in neighborhoods of different sizes. All houses in a given neighborhood are connected via the low-voltage grid to one substation of the DSO. The dataset is comprised of aggregated 15 min intervals of electricity consumption and production of 2928 homes from 44 substations in Belgium.

Our objective is to apply the proposed methodology, specifically the two-level clustering algorithm, to detect the consumption and production profiles considering the customer behaviors in order to contribute to future decision-making problems in this field.

The proposed methodology was used through the application of the two-level clustering approach to Belgian energy consumption and production data. Daily consumption and production profiles were obtained, considering global features such as total daily consumption or production, and local features such as hourly consumption or production, respectively. Whereas, prototypical consumers and prosumers were discovered considering global features such as total yearly consumption or production, and local features such as daily consumption or production, respectively.

The proposed methodology has the following advantages. It allows obtaining different clustering results by using different time granules. Moreover, it allows considering different clustering objectives. In the first level, only some general statistics about the data are required; while in the second level, the main objective is the similarity in time for identifying the consumption or production profiles. The approach also allows for dimensionality reduction via feature extraction in the first level; doing so the clustering algorithm becomes more efficient. It extracts a set of measures from the original time series; as such it is possible to obtain good results using the Euclidean distance, whereas this measure cannot handle the original time series directly. It uses a finite set of statistical measures to capture the global and local nature of the time series; thus, the computational efficiency of the clustering algorithms can be improved and the use of more advanced clustering algorithms becomes possible. Finally, our approach allows obtaining clusters with homogeneous consumption or production levels in the first level and clusters with the same profile in the second level. We believe this latter characteristic is appealing to test decision policies as it is crucial that the situations are considered to consist of typical situations that might occur in reality.