High Availability in Critical Infrastructure: Lessons from the Berlin Power Plant Outage

The recent fire at a Berlin power plant has dramatically illustrated how vulnerable our energy infrastructure can be. Thousands of households were suddenly without power, hospitals had to switch to emergency generators, and critical infrastructure facilities faced the challenge of maintaining their operations. This event has triggered an important discussion: municipal utilities and operators of critical infrastructure are currently intensively reviewing the high availability and fail-safety of their systems.

While many companies are only now beginning to address this topic, Flux Master already offers a proven solution for highly available control systems today – with all the advantages of modern automation technology.

Why High Availability is Essential for Critical Infrastructure

Critical infrastructures such as water and energy supply, hospitals, traffic control systems, or production facilities cannot afford to fail. A shutdown not only has economic consequences but can endanger human lives in the worst case. The requirements for modern control systems are therefore clear:

• Redundancy at all levels: Every critical component must be present multiple times
• Automatic fault tolerance: The system must autonomously detect and compensate for failures
• Seamless failover: Operations must not be interrupted when a component fails
• Continuous monitoring: Potential problems must be detected early
• Rapid recovery: After a failure, the system must be fully operational again quickly

The Challenge of Traditional Solutions

Conventional automation systems are often not designed for true high availability. Many solutions offer redundancy at the hardware level but fail at the software implementation. Typical problems include:

Complex Configuration: Setting up redundant systems requires deep specialized knowledge and is error-prone. Often, several days or weeks of engineering effort are necessary to configure redundancy correctly.

Incomplete Redundancy: Many systems offer redundancy for the controller but forget critical components such as databases, visualizations, or communication interfaces. A single point of failure is enough to bring down the entire system.

No Automatic Synchronization: With classic solutions, both systems must be configured identically manually. Changes must be made twice, which is error-prone and increases maintenance costs.

Long Switchover Times: Even with existing redundancy, switching to the backup system often takes several seconds or even minutes – an eternity for critical processes.

Flux Master: High Availability by Design

Flux Master was developed from the ground up with a focus on high availability. Instead of treating redundancy as an afterthought, it is an integral part of the system architecture. This leads to decisive advantages:

Automatic Cluster Formation

Flux Master instances automatically recognize each other on the network and form a highly available cluster without manual configuration. Synchronization occurs automatically in real-time – configuration changes, program updates, or database entries are immediately replicated to all cluster nodes. This eliminates human error and drastically reduces engineering effort.

Complete System Redundancy

Unlike many other solutions, Flux Master offers not only redundant controllers but also:

• Redundant Databases: All process and historical data are automatically replicated across multiple nodes
• Redundant Visualizations: Web interfaces and HMI applications remain available even if a server fails
• Redundant Communication: All fieldbus and protocol connections are mirrored
• Redundant Edge Computing Functions: Data processing and AI algorithms run in parallel on multiple instances

Sub-Second Failover

Flux Master continuously monitors the status of all cluster nodes via a high-performance heartbeat mechanism. If a node fails, the system automatically fails over to a functioning node within milliseconds. This switch is invisible to the user – no restarts, no connection drops, no data loss.

Shared Hardware Connection

A key advantage: both redundant instances use the same EtherCAT IO system. This means the connection to sensors, actuators, and field devices remains intact even if the master instance fails. The failover time is limited to pure software switching – hardware communication continues without interruption.

Investment in the Future

The Berlin power plant outage is a warning but also an opportunity. Companies and authorities that now invest in highly available control systems not only protect themselves against future failures but also position themselves as reliable partners for their customers and citizens.

Flux Master makes high availability affordable and easy to implement. While others are still thinking about redundancy concepts, forward-thinking operators are already using a proven solution that combines high availability, cybersecurity, and modern cloud technology.

Conclusion: Be Prepared Instead of Having to React

The time for reactive action in automation technology is over. Operators of critical infrastructure must act proactively and design their systems for high availability before a failure occurs. Flux Master offers exactly the solution needed for this: mature, tested, and proven in productive use.

While other vendors offer complex and expensive solutions that require months-long implementation projects, Flux Master can be put into operation within a few days. Automatic cluster formation and complete redundancy of all system components make it the first choice for companies that cannot compromise on availability.

ootb automation GmbH is happy to support you in the evaluation and implementation of highly available automation solutions with Flux Master. Our expert team advises you on architecture concepts, conducts feasibility studies, and accompanies you from planning to successful commissioning.

Contact us today to learn how we can take the availability of your critical systems to a new level. Because the best time for high availability is always: now – before the next outage comes.

How Agile System Development Creates Scalable Automation Solutions

In modern industrial automation, companies face the challenge of developing control systems that are both powerful and cost-effective. This case study demonstrates how ootb automation GmbH enabled a machine manufacturer to build their automation system incrementally through agile development methods, while maintaining complete cost control.

The Initial Situation

A mid-sized manufacturer of specialized machinery faced a typical challenge: the company needed a modern control system for a newly developed production machine. The particular difficulty was that system requirements would grow over time. Initially, the machine should be launched with basic functionality to generate first sales. As demand increased and customer feedback accumulated, additional features would be added successively.

The classical approach would have required specifying and developing the complete system from the start – an approach associated with high upfront investments and the risk of developing features that ultimately wouldn’t be needed. The customer therefore sought a more flexible path that would allow the system to grow organically.

The Agile Solution Approach

ootb automation proposed a consistently agile development approach. Instead of comprehensive upfront planning, the project was divided into clearly defined development phases, with each phase creating immediate value for the customer. This iterative approach enabled prioritizing features based on actual needs and aligning investments directly with business success.

The development team consistently focused on the question: “Which feature brings the greatest benefit to the customer in this phase?” This created a development roadmap closely oriented to real market requirements.

Phased Implementation

Phase 1: Building the Foundation The first step implemented solid control logic representing the machine’s core functionality. The Human Machine Interface (HMI) was deliberately kept simple and connected via the established Modbus/TCP protocol. This solution was technically mature and enabled rapid market entry.

Phase 2: Transparency Through Data Acquisition Once the first machines were in operation, the need for better production process traceability became evident. The system was extended with web server-based data recording that automatically generated Excel reports. This enabled customers to perform detailed production analyses for the first time.

Phase 3: Modernizing Communication As system complexity grew, the Modbus/TCP protocol reached its limits. Visualization was migrated to the more modern MQTT protocol, enabling significantly more flexible and performant data communication. The modular software architecture allowed this migration without reimplementing core functions.

Phase 4: Optimizing Production Planning The machine’s success led to increasing production volumes. In this phase, intelligent planning and control systems were integrated, enabling optimal utilization and higher throughput. This investment paid for itself immediately through increased sales figures.

Phase 5: Location-Independent Access Increasing connectivity in industry enabled another evolutionary step: the development of a modern web interface accessible from both the shop floor and the office. This significantly improved usability and enabled new use cases.

Phase 6: Predictive Maintenance As the culmination, the system was extended with intelligent sensor data analysis. By analyzing wear indicators, predictive maintenance could be implemented, minimizing unplanned downtime and significantly increasing Overall Equipment Effectiveness (OEE).

Measurable Success

The agile approach proved to be a complete success. The customer maintained full control over budget and functionality throughout the entire development process. Each development phase was only pursued when justified by concrete market requirements. The modular software architecture enabled cost-effective adaptation of interfaces and protocols without affecting existing functionality.

The financial aspect was particularly remarkable: instead of a large initial investment, the customer could distribute development costs over several years. Each new expansion stage paid for itself through the additional sales or efficiency gains it enabled. The risk of bad investment in unneeded features was completely eliminated.

Conclusion: Agile Development as a Success Factor

This case study impressively demonstrates how agile methods can play to their strengths even in industrial automation. The ability to grow systems organically while always keeping maximum customer benefit in focus creates a decisive competitive advantage.

ootb automation GmbH has comprehensive expertise in simplifying complex automation projects. Our range of services includes risk-free simulations for concept validation, real-time monitoring of critical processes, remote maintenance for rapid problem-solving, and development of reusable software components that sustainably reduce your development costs.

Through our consistently agile approach, we create solutions perfectly tailored to your requirements that grow with your company. Contact us to learn how we can make your automation project a success.

Revolutionizing Stage Automation: Enhanced Flexibility with Reduced Complexity

The automation of modern theater stages demands the highest standards of precision, safety, and flexibility. Every stage is unique, and every production brings new technical challenges. This case study demonstrates how ootb automation GmbH developed a revolutionary system for a renowned theater operator that drastically reduces complexity while maximizing flexibility.

The Challenge: Custom Solutions as Cost Drivers

Theater stages are among the most demanding environments for automation technology. The technical equipment must meet the highest safety standards while not restricting the creative freedom of directors. Our client, an established provider of stage automation solutions, faced a fundamental problem: every new installation required extensive custom programming.

The consequences were severe: long planning phases delayed projects, development costs rose continuously, and error susceptibility increased with each new development. Each stage had its own peculiarities – different numbers and types of line sets, turntables, lift platforms, and traps. The previous solution consisted of developing software from scratch for each project or laboriously adapting existing code.

The client recognized that this approach was not sustainable in the long term. The vision was clear: a system had to be created that enables flexible configuration without requiring custom software development for each installation. Theater technicians should be empowered to configure and adapt systems independently – without depending on external programmers.

The Solution Approach: Thinking in Reusable Building Blocks

ootb automation analyzed the problem and developed a fundamentally new approach. Instead of writing individual code for each stage, a universal software platform should be created based on the concept of reusable components. The core idea: all typical stage elements – whether line sets, motors, frequency converters, or safety devices – should be available as preconfigured software modules that can be flexibly combined.

This vision required fundamental rethinking of software architecture. The system had to be able to create virtual representations of physical components, intelligently connect them, and automatically consider all safety-relevant dependencies.

The Implementation: RIO – Reusable IO Framework

After two years of intensive development, the time had come: RIO (Reusable IO Framework) went into operation. This innovative platform enables the creation of virtual control cabinet components that communicate with each other via standardized bus systems. Each component has its own configuration interface, allowing parameters to be adjusted without programming.

The Centerpiece: Intelligent Component Library RIO features an extensive library of predefined components. A hoist chain, for example, is available as a complete software object that already includes all typical functions: position control, limit monitoring, load monitoring, synchronization with other hoists, and emergency behavior. Users select the required components and connect them via drag-and-drop.

Flexible Axis Configuration A particular advantage of RIO is the autonomous control of individual axes. Each mechanical axis – whether a curtain track, turntable, or lift platform – can be configured and controlled independently. At the same time, the system allows complex synchronizations: multiple hoists can be grouped together, movement sequences can be choreographed, and safety interlocks are automatically considered.

Practice Meets Theory RIO’s true strength is revealed in everyday project work. What previously required weeks of programming can now be configured in days. A new line set? Simply select the corresponding component from the library, assign the I/O channels, enter parameters such as speed and acceleration – done. The system automatically generates all control logic including visualization and diagnostic functions.

Measurable Success: Time and Cost Savings

The introduction of RIO led to a dramatic improvement in project efficiency. Programming and commissioning times were reduced by an average of 60 percent. Errors arising from individual programming are largely a thing of the past, as the reused components have already proven themselves in numerous installations.

The flexibility gained during project execution is particularly valuable. Changes that previously required extensive recoding can now be realized by adjusting the configuration. This is of immense importance in theater practice, where requirements frequently change during the installation phase.

Reusability also extends to functional enhancements. New features developed for one project are automatically available as components for all future projects. This creates a continuously growing toolkit that becomes more powerful with each project.

Client Testimonial from Practice

Tiroler Festspiele, a long-time user of the technology, confirms its practicality:

“We have set up many productions with the Stage Automation system and value both its reliability and customer support. The ability to make adjustments ourselves without having to wait for external programmers every time has significantly improved our workflows.”

Tim Herold, Technical Director, Tiroler Festspiele

Technological Expertise at ootb automation

This case study exemplifies the core competency of ootb automation GmbH: the development of intelligent automation platforms that master complexity without transferring it to the user. Our range of services includes complete PLC software development, design of reusable automation platforms, and comprehensive consulting in optimizing automation processes.

The RIO framework represents our approach: through intelligent abstraction and modularization, we create systems that are both powerful and manageable. Theater operators can focus on what matters – the artistic production – while the technology works reliably and flexibly in the background.

If you also face the challenge of making complex automation systems more efficient, contact us. We analyze your specific situation and develop customized solutions that sustainably optimize your processes.