Written by Geert Meulenbelt & Léopold Shaabani Ardali
Debunking some myths
The word Optimization is used everywhere: from search engines to manufacturing processes. In particular when using optimization in relation with data the term is often used in a
n inappropriate manner.
This post firstly defines four optimization types, will subsequently provide some insights into multidimensional optimizations and concludes with a set of useful takeaways.
Four optimization types
The graph below displays four different types of optimization based on their complexity and potential impact on ROI, showing the extraordinary ROI potential of multi-dimensional optimizations.
Business Intelligence packaged as optimization
Many solutions that talk about optimization do not optimize anything at all. A lot of effort is spent collecting and cleansing data and making fancy dashboards on top. Creating insights is certainly helpful, and working on data and its governance is a prerequisite to become more data driven. But all we are talking about here is Business Intelligence that creates fairly limited value beyond a first aha-moment when data is visualized for the first time (and you suddenly realize that the data and/or the underlying business rules are wrong).
Establishing business rules on a particular optimization challenge often helps to simplify problems and pick the so-called ‘low hanging fruit’. These written rules try to mimic the decisions an expert would take to handle the process efficiently, such as “if consumption lowers, then lower production” or “First, ignite the gas boiler (cheaper), then, if not enough, the electrical boiler.”
This approach, that belongs to the “Expert systems”, boomed in the 1980-1990s in the finance and medical industries. However, several drawbacks make them not very efficient:
- their rigidity (if the system for which the rules are written change, the rules must themselves change),
- their complexity (to mimic accurately an expert, it might be needed to handle hundreds of rules, in a precise order),
- and their inability to handle efficiently too complex decisions (e.g. decisions where tradeoffs must be made between several equivalently appealing decisions a priori – simple example : should I start two small boilers at 75% capacity each, a small one at 100% capacity and another one at 50% capacity, a large one at 100% capacity or two large ones at 50% capacity?),
Single dimension optimization
The large majority of companies providing optimization solutions powered by algorithms do so with a single category or dimension: optimize people in a planning for instance, or optimize electricity consumption on machines in a production process.
However, optimizing a single dimension enables cost savings on this particular dimension, but could well degrade other dimensions. Consider a factory cluster that must meet a required total production target and deliver this production to customers across a whole area:
- A production planning tool will consider only production costs. This will lead to the specialisation of every factory in its most efficient production. However, this will increase delivery costs.
- A logistics planning tool will only consider deliveries. Hence, clients will always be delivered by the closest factory. However, to do so, this might require rescheduling the production in smaller batches, since each factory will produce many different specialised products. This process could also be costly.
The more dimensions you can take into account, the more powerful and rewarding an optimization will become. Or putting it differently, the closer you can bring digital technologies to tackle real world situations, the better outcomes you will get.
Both classical (operations research methods) and machine learning techniques (such as the famous example of the 40% reduced data center cooling bill by Google) are used to solve these problems. These techniques include reinforcement learning. In a broad sense, reinforcement learning is concerned with how software “agents” ought to take actions in an environment in order to maximize some notion of cumulative reward. This agent, when facing several instances of the problem studied, will gradually learn how to take decisions that maximizes its total revenue (or minimize its total costs) based on the experience it acquired.
The trick is to be able to combine these different techniques at different levels. And that is exactly the holy grail of multi-dimensional optimization which is the playground of only a minority of the solution providers. Although we are just starting to see the results, ROI is often an order of magnitude higher than anything achieved on a single dimension.
Characteristics of multi-dimensional optimization
The gap between Excel spreadsheets and complex multi-dimensional optimization is huge. Since few solutions exist in the market, building a viable solution internally is even less of an option due to among others time to value, quality of outcomes or maintainability. Since there are so many elements to put in place (from data ingestion and quality and data governance to data visualizations targeted to operational users) it is highly recommendable to consider optimization and the underlying engine as a component that should be sourced externally, at least in short term.
A clear priority should be given to solving business problems and proving ROI.
To get this point, a number of characteristics on optimization need to be better understood.
Understand the needs of stakeholders
Multi dimensional optimization looks by definition at multiple dimensions. This means that optimization in many use-cases will create different scenarios with people selecting a particular scenario. And each scenarios takes into account a mix of elements.
Different stakeholders have different requirements:
- Simulations (what-if) questions are often asked by senior management on a low-frequency basis. These are used as an input for strategic planning: what happens if my CO2-costs skyrocket, what would be the best location of a new plant or warehouse? Operational managers ask similar what-if questions but on more down-to-earth topics: how will a new supplier impact my supply chain, what will be the impact of a new injection point on my network? It’s all about providing information to enable the right decisions by humans.
- Optimizations are used for operational teams and typically on a more frequent (often daily) basis: given the availability of assets and the constraints, how should resources be allocated? The results of the selected scenario are fed into operational tools.
- Alarms should trigger actions (alternative scenarios) to know what options are in play to make the best decision. There can be external events (a flood on a river that blocks transportation) or internal events (an employee being sick). Decision making based upon scenarios of different options tends to have a sense of urgency and are closer to real-time.
- Simple visualizations are often highly appreciated by stakeholders that are farther away from the “core of the action”: be it internally (admin, finance) or externally (regulators, customers).
Think as a Network
Optimization challenges can be found anywhere in the value chain. Take for instance the production and distribution of a good: the production line (energy, CO2, urgency, HR…), final goods in a box (size, conveyor speed), boxes on a pallet (size, sequence)… An easy way to grab the concept is to look to this as a network with different nodes and different flows between nodes with constraints everywhere.
Especially with flows, creating a dynamic digital representation of a network (a so-called Digital Network Twin) is complex and requires know-how on how flows behave within a network over time.
Once the way optimizations are designed are known, a model on the overall topology of the network can be created. Based upon this model end-users are free to create specific variants or add nodes. This brings end-user autonomy which is an important ingredient for success.
Naturally such a network model needs occasional fine-tuning but field feedback suggests that yearly updates are sufficient.
It is important to understand that problems that initially look very different can be solved in a very similar way when looking at them as a network of flows constrained by business rules. An example of a similar use-case from an optimization point of view:
- The optimization of the connection of new biogas production plant to the global network,
- The optimization of the location and dimensioning of a new warehouse in a freight logistics network.
Contrary to simple single dimension problems, solving the difficulties and constraints of real-world problems is much more difficult. It is easy to understand why: when more dimensions are taken into account, the number of possible solutions grows exponentially.
Therefore hoping to find the best solution by simply enumerating all possible outputs and picking the best one is a naive and unrealistic dream that gets more and more out of reach when the size of the problem increases.
Generally, powerful algorithms exist for simply-formulated one-dimensional problems, such as the travelling salesman problem for the daily fixed-customer visit described above. However, as soon as other non-trivial dimensions are added, these algorithms fail and no “off-the-shelf” solution exists anymore.
Finally, the constraints and related business rules may vary over time (or are overlooked at the beginning), with potentially great impacts on how an optimization is designed. A small change in a business rule can have huge impacts on optimization.
There is no magic button to make things work
Sometimes you hear people that swear by a particular technology “if you don’t use deep neural networks, then you are not really optimizing”. This totally misses the point: there are always different ways to reach results. It’s the quality of the outcome and the efficiency to get there that matter, not the underlying technology.
Also forget the dream of a magic all-encompassing algorithm (with the associated Intellectual Property rights of course) that you can create by pushing a button and put in production to solve a particular optimization problem.
The reality looks more like this:
- Before carrying out any optimization, a model should be build that defines how different nodes and constraints interact with each other. This model or topology is adjusted to take into account a particular situation.
- There is not a single, unified algorithm but a multitude of decentralized algorithms that together produce an output (a flat file) based upon a particular situation (a model) at a particular point in time.
- Different algorithms are doing different things at the same time. In a heating network you need both the demand forecast and the propagation of flow and temperature in the pipes to get meaningful results.
- The algorithms are dynamic and subject to frequent change.
The conclusion is that real-world problems are unique and optimization challenges need to be ‘cracked’ on a case-by-case basis. Having the brains and tools to do that is a prerequisite.
To put it bluntly: optimization is not an algorithm or a technology, it is a profession.
And this implies also that there is a process and a multidisciplinary team whereby you test and learn different optimization options. In practice such an agile way of working often conflicts with traditional “Gantt-type” project approaches that are used in large companies.
Optimization accounts for less than 25% of the time within a project
Getting operational business knowledge is a mandatory first step: clearly define KPIs, set priorities and align the problem with a subset of algorithms that may be used. It is far from clear at the beginning of an effort to determine what really influences the results of an optimization.
Getting and understanding the right data often represents at least 50% of the time spent in a project. As always the “rule garbage in = garbage out” applies. Time can be saved by automating smart ingestion in particular for time series.
Finally, one should never underestimate the data-engineering complexity due to the difference between an initial test cycle on a few historic datasets and an industrialized solution with IT and security involvement to connect data streams.
In the end it’s all comes down to trust
Trust and acceptance of the outcomes of optimizations depend both on the capacity to explain these outcomes (technically and visually), business autonomy (by adjusting and selecting scenarios) and the willingness of an organization to invest in change management. These elements are often of a much bigger importance then the optimization itself. It’s people that will make the difference.
Conclusions & Take aways
Understand the needs of stakeholders from strategic simulations and operational optimizations to visualizations. Think in scenarios with operational teams in charge to select the best scenario.
Think as an end-to-end Network with multiple optimization components that are linked and can be presented as a dynamic Digital Network Twin.
Don’t believe in generic, automated solutions: complex and high ROI optimization needs to be ‘cracked’ on a case-by-case basis. Smart brains and the right tools make a huge difference.
Understand that results are obtained gradually by agile teamwork, with evolving constraints and business rules having a potentially large impact.
Even though huge value can be created by optimization engines, the optimization component itself represents often less then a quarter of the time spent in projects.
Trusted outputs are key to success and require investments in change management and high-level management endorsement.