Data management in the cloud: problems and solutions
In times of digitalisation, well thought-out data management is becoming more and more important, but at the same time increasingly demanding. The sheer volume of data is growing at an enormous speed. Some of this data is very important, but a large part is also partially irrelevant.
In addition to the large volume, the situation is also complicated by the fragmentation of data: within a company as well as within the data infrastructure. Different parts of a company often manage their own data instead of storing it centrally. Best practices, such as master data management, are often only applied selectively and, as experience shows, are poorly integrated into work processes.
In addition, the emergence of cloud applications has changed the situation significantly. By storing data in different data centers, possibly even worldwide, a multitude of regulatory requirements must be observed. One example in this context would be the storage of critical corporate, personal or tax data. However, the cloud can not only be part of the problem, but also the solution.
What is data management?
The Data Management Association (DAMA) defines data management as “the processes used to plan, specify, activate, create, capture, manage, use, archive, retrieve, control and delete data.
In practice, data management is all about making the handling of corporate data more efficient and ensuring that your organization meets a variety of legal and regulatory requirements and that data is consistent. Within the data management thus defined, there are several sub-areas that require special attention: data protection, governance and data intelligence.
Data Protection
In recent years, data protection has become increasingly important. Apart from the much discussed area of personal data protection (i.e. collection, storage and deletion of personal data), this refers to the general protection of data. The question is how a company can protect itself against loss or damage of data, no matter whether it is archived data sets or daily updated business data, in an increasingly confusing data situation.
Data hygiene is an important keyword in this context. The internal IT team is not only involved in the technical aspects, but can also significantly increase data hygiene by training employees in the correct handling of data.
The protection of data is rounded off by a good overview of the accruing data and planned measures for its backup and recovery. Helpful here is also a so-called Disaster Recovery Plan (DRP) which takes effect in case of emergency and additionally protects you.
Governance
The topic of data governance can also be divided into two areas: International data governance describes legal and global regulations for the exchange and management of data between individual countries, particularly with regard to country-specific data law.
Internal data governance is concerned with data consistency and compliance with legal regulations within the company. This is less about data protection than about data consistency, traceability and optimization of data flows and processes. Good cooperation between legal and technical departments is particularly important in the overall data governance process.
Data intelligence
Data intelligence and data hygiene belong firmly together. Because only if my data is reliable I can use it for further analysis. Compared to the two previous topics, Data Intelligence is more oriented towards the future. Here it is all about drawing the right conclusions from existing data and metadata and learning, for example by using machine learning and AI.
Data Intelligence can help, for example, to develop predictive maintenance systems or to identify and exploit market movements at an early stage. A lot has happened here in recent years and companies are increasingly relying on the meaningful evaluation of their data to identify potential and risks across all areas of the company.
Challenges through the Cloud
The emergence of as-a-service functions has added several levels of complexity to data management. Even before the increased use of the cloud, enterprise data was widely distributed. The cloud then adds SaaS services, IaaS/PaaS workloads, and features like VMware Cloud to the various data layers that already exist, creating even more storage locations. Ironically, the ease of use of SaaS has often made IT work more complicated while benefiting the end user.
But how do you maintain control and resiliency when things have spread everywhere?
Solutions: technical & organisational
It is clear that the additional data points in the cloud must not be an obstacle to using them. Rather, it is now a question of developing systems to reduce complexity. This is simply not possible without the use of cloud-based technology.
This requires a holistic approach to managing data across endpoints, infrastructure and cloud apps. By centralizing visibility and control over enterprise-wide data, data risk and the cost of data protection is significantly reduced.
Whether internal systems are developed and implemented or existing data management tools are used depends on the individual company with its different requirements and must be evaluated on a differentiated basis. IT companies should find it much easier to manage their data themselves than others. For these companies, external solutions would then be more appropriate.
But no matter which system is used for data management, a functioning communication within the company is particularly important. This includes IT, as well as the individual employee or the legal department.