Forecasting in a pandemic

 

Forecasting systems and the algorithms that support them have evolved a lot in the last few years. Nowadays, more and more components in the Cloud help us or give us the ability to all kinds of users and developers to build simpler algorithms that can achieve an exciting output and add value to organizations.

 

On the other hand, with more and more external data at our disposal, third party data, it is possible, in real-time, to have access to variables that add a lot of value to the forecast algorithms, with constant updating, and that are fundamental for a reliable prediction of future events.

 

 

There is a vast amount of Machine Learning solutions, an ecosystem that is sometimes confusing for those who do not follow these topics so closely - Computer Vision, Anomaly Detection and Forecasting, are some examples.

 

Forecasting is by far the most sought-after solution today by companies. It is fast to implement, easy to validate results, with excellent reliability indices; it is a fundamental tool to help decision-making in organizations. An ally for managers since it allows them to quickly and in real-time have access to forecast, patterns, and consumption behavior. This is, without a doubt, an excellent first approach and a first step towards implementing Artificial Intelligence in companies.

 

There are many algorithms available to Data Scientists, from the more classical approaches, which are based on doing linear regressions to identify the trend of the data, algorithms that are focused on time series and can make very long term forecasts, more complex algorithms with components of trend forecasting, seasonal variations, including impacts of holidays and feast days, which, with the addition of external variables, are more complex, but also more resilient and reliable.

 

All this, driven by the processing power that the Cloud has brought us, which is now available to everyone and has "democratized" the access and implementation of this technology. These cloud solutions, with pre-configured algorithms, can also, to some extent, cover the general needs of most organizations and put artificial intelligence a short distance away from the actual scenario of organizations. Artificial intelligence and its predictive capabilities are no longer a far-off, science-fiction vision.

 

 

However, the last 12 months’ scenario has added a dynamic and abrupt change in time series and a new layer of complexity to the subject.

 

There are several use cases, for example, in a sector such as catering that was brutally impacted by the pandemic, wherefrom one day to the next, a definite trend and a very constant pattern of consumption went to zero consumption, causing disruption in this technology and the forecast algorithm.

 

The pandemic context thus posed considerable challenges to data scientists. This total data disruption quickly made the learning patterns of the algorithms obsolete, as they could no longer be applied. Abnormal situations, therefore, arose, such as pessimistic predictions, because the time series could not adjust to the absence of data. The solution went through a creative exercise and a lot of work, trial and error, the search for new external variables, macroeconomic variables, indicators previously disregarded, and an increasing complexification of the algorithms. The introduction of statistical variables, formulation of hypotheses, removal of missing values, data interpolation, among other techniques, were the basis of the work in recent months.

 

A "chaotic" scenario, but at the same time exciting, for those who work in this area, where the variability of each company's activity was added, even if they operate in the same sector of activity. Companies that closed down and stopped altogether, others that remained with minimal activity, still others that changed between one regime and another over time. Selective shutdowns, changing scenarios, legislative and rule changes between weekdays and weekends, weekly and bi-weekly changes, truly atypical and uncorrelated consumption patterns, or even the change in customers' consumption habits.

 

The forecast algorithm is versatile and moldable. It can be enriched with external information or other approaches. This experience and period have also allowed us to empower the teams and professionals in the area with different skills and additional agility. The response to this scenario requires a very strong preparation and research, realizing that the last recent temporal period, being a really atypical period (so we all wish), will force us to make decisions about the "weight" to consider of this period, in relation to the pre-pandemic periods and, understand, how this variation behaves in reality.

 

Since it is impossible to predict how the "new normal" will be in terms of consumption habits and the interactions of consumers and customers with companies, the big question is whether the forecast algorithms will continue to have this need to adjust for a long time, or whether they will be able to adapt quickly to new trends and new dynamics. Resilience will be a critical factor that all organizations will have to pay more and more attention to. A period as disruptive as this one has, in fact, put all the players involved to the test.

 

The trend points to the need for increasingly personalized and customized solutions, dropping the concept of "auto-ML" or "one size fits all" solutions. Customization will be a crucial factor.

 

These have been incredibly demanding times for those who work with data, forcing Data Scientists to "get out of the box" and look for different formulas to achieve the ultimate goal: to provide business decision-makers with reliable data to support their decision-making!

 

Published (in Portuguese) in Dinheiro Vivo