Total control: real-time analytics with Monitoring 2.0

A lot helps a lot – but only if it happens fast enough. This is exactly where fast data scores big: the combination of big data and real-time evaluation enables completely new approaches. For example, Monitoring 2.0, for which Eddie is project manager and product owner at ING Germany – one of the many exciting IT roles at this digital challenger bank.

What is the project about? Eddie uses fast data to monitor IT systems, e.g. status, CPU resources, memory and process stability. Log files are continuously evaluated to improve system operation. The huge amount of data is analyzed so quickly that immediate reaction is possible when a relevant event occurs. The goal for the future: an end-to-end solution for near-time monitoring and predictive maintenance.

Monitor, detect, predict

Exasperation due to a sudden system failure – every IT specialist knows this feeling. This was Eddie’s approach: "We had the following scenario. A developer wanted their program to go live before release, and the tester was testing it accordingly. But when a server failed, the tester didn’t notice. There was no way to know what the problem was. So we introduced Monitoring 2.0 – to provide an environment where all relevant stakeholders have an overview of the system status and other details. In the past, it often took several days to investigate the cause of failures. This time is now reduced to minutes or hours." System failures can even be predicted and prevented. Eddie continues: "Our next step is to carry out predictive machine learning. This will then make it possible, for example, to anticipate that a system could fail in a few days." Thanks to the large database, trained models can then recognize patterns and derive correlations that a human observer would not notice, and certainly not so quickly. Thanks to machine learning, in the future it will be possible to predict whether a particular operation will lead to problems even before the operation has been carried out. This is predictive maintenance at its best.

Pioneering architecture for the ELK stack

However, the implementation of this ambitious project requires a special architecture, which has to be specifically designed. So the IT experts at ING had to break new ground. Due to banking-specific regulatory requirements, using conventional cloud solutions was not an option. Another challenge was access to legacy systems – an important aspect for a large corporation like ING. The solution is a combination of “normal” world and cloud world. "We monitor all worlds together on one dashboard," Eddie points out. The creative architecture therefore consists of a clever orchestration of suitable components, many of which run in the ING Private Cloud. Apache Kafka handles the exchange of data. The advantage of this event streaming platform is that it can process data from many different distributed applications and sources at once, and very quickly – in the range of milliseconds. Kafka offers high system stability and also integrates legacy systems such as traditional databases and middleware. The monitoring processes run in the ELK stack, which connects the three open-source components: Elasticsearch, Logstash and Kibana (also called Elastic Stack). Logstash acts as a pipeline that pulls data from Kafka as needed, thus conserving ELK resources. The data analysis itself is carried out by the search engine Elasticsearch, which uses fast NoSQL indexing while also supporting high availability and scalability through parallelization and a distributed structure. The ELK stack is rounded off by Kibana, a dashboard for result visualization. In addition, the application environment Jupyter Notebook is integrated for the use of machine learning.

In-house solutions, agile development

Not only the architecture, but also many components had to be developed by ING's IT experts. For example, an S3 storage solution for archiving. The advantage, according to Eddie, "Our S3 development prevents Kafka's performance from being slowed down by too much data storage. Data can then be deleted after one week on Kafka, for example." Many new interfaces (connectors) also had to be programmed to enable data exchange between the components. Eddie is also currently working on a leaner version of the analytics environment without Jupyter. Of course, Eddie does not do all this alone. He leads several teams in which the colleagues – like all IT staff at ING – work together in an agile way. For example, the Fast Data Squad or the Project Squad for Monitoring 2.0. Depending on the topic, Eddie works with different agility approaches – sprints or Kanban. "When deciding on an approach, the squad naturally also has a say in the decision. In the beginning, we always first discuss how the team itself would like to work," adds Eddie.

Creativity is in demand: IT jobs at ING

Creative challenges, forward-looking areas of responsibility, agile work – these are just some of the advantages of an IT job at ING, which is constantly looking for new staff at its Frankfurt a.M. and Nuremberg locations. Plus, there is plenty of room for further personal development, a generous education budget and, last but not least, a worldwide IT community. As part of a global company, working at ING Germany always has an international dimension. "What we’re doing with Monitoring 2.0 is a first within the Group, also generally in Germany, perhaps even worldwide," Eddie claims. It is therefore planned to use it for other subsidiaries of the Dutch parent company. Eddie also needs additional personnel for future projects. For example, in terms of interface, Eddie runs the Chatbot Squad and is planning a natural language query via Chatbot and Natural Language Processing (NLP) as a further component for Monitoring 2.0. At the moment, this is still a dream for the future, but given the current dynamics of fast data and machine learning, it could soon become reality. If you are interested in participating in these exciting developments, you can find out more about IT opportunities at ING on our Career page.