Data protection framework for the development and use of AI systems in social institutions
Prepared by the Metaverse Europe Foundation with the kind support of the Gleiss Lutz data protection team.
Artificial intelligence (AI) opens up a wide range of opportunities for social institutions to make workflows more efficient, tailor individual support services more precisely and optimize administrative processes. One conceivable example is an early warning system for child endangerment: a social institution has anonymous reports on so-called risk assessments. An AI evaluates these reports and assists the specialists with a “third opinion” for risk assessment in new cases. Or the AI generates fictitious cases on which new specialists can be trained.
The use of AI systems is usually based on extensive databases, which often contain personal information. This brings data protection law to the fore.
The General Data Protection Regulation (GDPR) forms the binding legal framework for the processing of this data. It obliges social institutions to comply with a large number of specific requirements when developing and using AI systems. Certain institutions are also subject to other data protection requirements, for example under social or church law.[1] This white paper provides a practical overview of the central data protection requirements that institutions must observe in connection with the development and use of AI systems.
[1] The detailed presentation of these special legal requirements is not the subject of this white paper, which is only intended to provide an initial overview. However, the content of the requirements is so similar that the requirements of the GDPR can also serve as a guide for institutions that are subject to other legal regimes in addition or instead.
Clarification of responsibility under data protection law
If institutions want to use AI systems, a central question should be at the beginning of any data protection consideration: Who is responsible under data protection law for the data processing that takes place during the development and/or use of the AI system? This clarification is far more than a mere formality – it forms the basis for all further data protection considerations. This is because the applicable obligations and requirements can only be determined with legal certainty once it has been clearly established whether an actor is acting as a controller, processor or, where applicable, as a joint controller.
The GDPR makes a fundamental distinction between the “controller” and the “processor”. The controller is the body that decides on the purposes and means of processing personal data. It is the central addressee of all data protection obligations. It must comply with all material data protection requirements and be able to prove that a technical and organizational data protection system appropriate to the risk has been implemented. The processor, on the other hand, acts exclusively on behalf of and according to the instructions of the controller, i.e. not according to self-determined purposes.
In practice, this means that if an institution operates an AI system exclusively for its own purposes on its own servers, it is usually the sole responsible party. However, it is more common for several parties to work together – for example, if an institution uses an AI system from an external provider as a cloud solution hosted by the provider. In this case, the institution is the controller, while the provider usually acts as a processor. In such constellations, it is important to conclude a data processing agreement that clearly regulates the respective rights and obligations.
If providers of the AI system use the data collected as part of the application not only on behalf of, but also for their own purposes, such as for the further development of their AI systems, this can lead to what is known as “joint responsibility”. This means that both parties jointly decide on the purposes and means of data processing and are therefore also jointly responsible for compliance with data protection regulations. Such a constellation requires a transparent contractual agreement between the parties involved in which responsibilities, information obligations and responsibilities are clearly regulated.
Finding the relevant legal basis
Once the responsibilities under data protection law have been clarified, data controllers should consider the relevant legal bases. This is because the processing of personaldata (such as names or addresses and data that is linked to a person via such “identifiers”) is only permitted if a legal basis allows this – i.e. either the data subject has given consent or a legal provision permits data processing.
The development and training of an AI model can generally be carried out with anonymized data. Such data is not subject to data protection law, meaning that no permission under data protection law is required for its use. But beware: according to the supervisory authorities, the process of anonymization also constitutes the processing of personal data, which therefore requires a legal basis. Depending on the constellation, different legal bases come into consideration for different actors:
Public authorities: Public interest (Art. 6 para. 1 lit. e GDPR in conjunction with federal or state legal basis);
Service provider (e.g. for child and youth welfare services): Basis required under social data protection law.
When processing sensitive data (health, genetics, sexuality, religious affiliation, ideological beliefs, political opinions, etc.), an exception pursuant to Art. 9 para. 2 GDPR must also apply. Insofar as no special legal regulations apply, only explicit consent (Art. 9 para. 2 lit. a GDPR) can generally be considered as a possible legal basis, although it is practically impossible to obtain this.
A legal basis is also required for the processing of personal data as “input” to an AI system . To find the relevant legal basis, a distinction must also be made here depending on the controller using the AI system in the specific case. If the AI system is used to optimize a previous task, the legal basis for the processing of personal data as “input” of the AI system can usually be based on the legal basis that legitimizes the data processing in the context of the performance of this task (without AI). This is because no special legal basis is required for the use of certain technical means or specific software for the same processing.
The following chart illustrates the different legal bases – depending on the type of data processing body – that can be considered for the processing of personal data:
Prohibition of automated individual case decisions
If an AI system is used to process personal data, particular care must be taken to ensure that no unauthorized automated decisions are made in individual cases. This refers to decisions that are made solely by the AI system without significant human involvement – and which also have the potential to significantly affect the data subject. This may be the case, for example, if benefits are refused or other detrimental measures are taken.
Automated decisions are generally not permitted. However, the GDPR provides for some clearly defined exceptions under which they may be permitted. An automated decision is only permitted if it is necessary for the conclusion or performance of a contract, if it is expressly permitted by a legal provision or if the data subject has expressly consented. The requirements are particularly strict when sensitive personal data, such as health data, is processed: In these cases, there must either be express consent from the data subject or a specific legal basis regulated by law in the substantial public interest. As both are (so far) very rarely the case, automated decisions based on sensitive personal data can hardly be implemented in a legally compliant manner.
In order to avoid unauthorized automated decisions, it must therefore be carefully checked before using an AI system whether there may be an adverse effect on data subjects in the respective use case.
If this is the case, it must be ensured that it is not the AI system that makes the decision, but a human being. And not just formally, but also in terms of content: the human must make the decision independently, based on the available facts and taking into account all legal requirements – and not externally determined by the result or suggestion of the AI system.
For this to succeed, clear and documented processes are needed to ensure that human decision-makers review and make decisions with the same care as without AI support. The contribution of the AI system may only lead to an improvement in the quality of the decision or speed up the decision-making process, but not anticipate it.
Duty to inform
According to the GDPR, data controllers are obliged to provide data subjects with comprehensive information about the processing of their personal data if they collect the data from the data subject themselves or from another source. In particular, the obligation to provide information covers the scope, purposes, legal basis and duration of the data processing as well as the rights to which the data subject is entitled under the GDPR.
However, the mere use of an AI system does not automatically lead to new information obligations under the GDPR. The AI system can merely be a technical aid for an existing processing activity, comparable to any other software used to process data. Institutions therefore do not have to provide additional data protection information to data subjects simply because they use AI.
The situation is different if personal data that was originally collected for a specific purpose (e.g. for the performance of a specific task) is now processed for the purposes of AI development. In this case, the processing purpose changes. Institutions are then obliged to inform the data subjects about this new processing purpose.
Additional information obligations may arise if the AI system makes automated decisions on a case-by-case basis. In such cases, the controller must inform the data subject that automated decision-making is taking place, provide meaningful information about the logic involved and explain the scope and intended effects of the processing.
However, it is not necessary to disclose technical details or the exact algorithm. Rather, the aim is to describe the process and the principles in such a way that the data subject can understand how their data was used in the automated decision. The information should be clear, transparent and easy to understand so that the data subject can comprehend the decision and its basis.
Rights of data subjects
When processing personal data for the purposes of AI development and with the help of an AI system, the rights of datasubjects to access, rectification, erasure, restriction of processing, data portability and objection must be observed – as with any processing of personal data.
The following special features arise in the context of the development and use of AI:
If the data subject requests information regarding automated decision-making, they must be provided with meaningful information about the logic involved and the scope and intended effects of such processing. It should therefore be possible for the controller to explain the functioning and the main processes of the AI system used in a comprehensible and transparent manner so that the data subject can understand how and why the decision was made on the basis of their personal data. To this end, the data controller may have to grant themselves the right to information from a third-party AI provider so that the AI system is not a “black box” for them.
If personal data is part of an AI model, the correction or deletion of this data usually involves considerable technical and organizational effort. Not least for this reason, it is advisable to use anonymized data for the development of AI models.
Insofar as the development of an AI model is based on a legitimate interest, data subjects have the right to object to the processing. The extent to which the controller can demonstrate compelling interests and reject the objection depends on the individual case.
Carrying out a data protection impact assessment
If institutions use an AI system that processes personal data, a data protection impact assessment (DPIA) must be carried out in advance in many cases. This is particularly mandatory when new technologies are used and there is a likely high risk for data subjects. According to the data protection supervisory authorities, this is regularly the case when AI systems are used that work with personal data.
The aim of the DPIA is to identify risks to the rights and freedoms of data subjects at an early stage and to take appropriate protective measures. The DPIA is therefore a structured procedure with which the controller becomes aware of the potential impact of the planned data processing.
A DPIA should contain at least the following points:
Description of the processing: What data is to be processed and for what purposes? What interests does the controller pursue?
Assessment of necessity: Is processing on this scale and in this way actually necessary to achieve the intended objectives?
Risk assessment: What are the risks for the persons concerned?
Risk minimization measures: What technical and organizational protective measures are planned to reduce or completely eliminate these risks?
Further obligations of the controller
When AI systems are used, personal data may be transferred to countries outside the European Union or the European Economic Area . This applies in particular if non-European providers or service providers are involved in the development or operation of the AI system.
In this case, it must be checked whether a so-called adequacy decision of the EU Commission applies to the data recipient (e.g. for Canada, Japan, South Korea). If this is not the case, so-called standard contractual clauses can be concluded. However, it should be noted that in some areas, particularly in the public sector, stricter requirements apply that expressly prohibit transfers to third countries without an adequacy decision.
If the development or use of an AI system leads to changes in the controller’s processing activities, this must also be reflected accordingly in the processing directory.
Summary
The use of AI in social institutions offers great potential for improving processes and optimizing individual support services. At the same time, the processing of personal data in the context of AI systems entails extensive data protection challenges. Responsible institutions must therefore carefully examine how they can meet the requirements of the GDPR. Clear responsibilities, the choice of the right legal basis, compliance with the ban on automated individual decisions and transparency towards data subjects are key success factors.
The following overview summarizes the most important measures that must be observed for the data protection-compliant use of AI:
No.
To-Do
1
Clarify responsibilities Who is the controller under data protection law? Is there only one or more controllers under data protection law? Are processors used?
2
Finding the legal basis Which legal basis is relevant for the processing of personal data?
3
Observe the ban on automated individual case decisions Are automated individual case decisions made by/with the help of the AI system? Is this permissible in exceptional cases?
4
Fulfilling information obligations Do data subjects need to be informed separately about the processing of their personal data? If so, how can this be ensured?
5
Fulfilling data subjects’ rights How can the rights of data subjects be guaranteed, in particular the rights to information, erasure and objection?
6
Data protection impact assessment Does a data protection impact assessment have to be carried out?
7
International data transfer Is personal data transferred outside the EU? If so, how is compliance with data protection regulations ensured?
8
Supplement the processing directory Is the processing of personal data already mapped in the processing directory or does it need to be supplemented?
Strengthen decision quality through data intelligence and immersive training Professionals in social institutions are faced with sensitive decision-making situations on a daily basis: Child endangerment, imminent need for care, crises in residential groups. There is a wealth of data available – case documentation, health data, progress reports – but it has hardly been used systematically to date. An example of […]
Inspired by Yoshua Bengio. Further developed by partners of the Metaverse Europe Foundation and experts “I never thought AI would become so powerful so quickly – and I’m worried about what could happen if we’re not smart about it.”These cautionary words come from Yoshua Bengio, one of the fathers of deep learning and winner of the Turing Award. In his […]
What does it even mean today to be “German”, “European” – or “free”? In a world in which global interdependence, technological acceleration and economic power structures permeate all areas of life, this question is becoming increasingly difficult to answer. Nation states are losing influence, while capital, algorithms and global markets are creating new realities. But freedom – genuine, self-determined freedom […]
On Wednesday, 26 March 2025, we met in the boardroom of the Berlin Capital Club for my first ‘AI Talk’ in a high-calibre setting. The speaker was Dr Eberhard Schnebel: philosopher, economist, theologian, professor of business ethics at Goethe University Frankfurt, board member of EBEN GR (European Business Ethics Network – Greek Chapter) and expert for data governance in Group […]
On 26 November 2025, we held our third pre-event for the AIM ETHICS Award in Brussels – a decisive step on our way to realising our award. Insight into the event Dr Reinhard Schultze from the Metaverse Europe Foundation opened the event with a fascinating introduction to the history of the AIM ETHIX Award. He impressively explained the inspiration and […]
The Launch of the Educational Metaverse On 30 October 2024, the new educational metaverse initiative of the Metaverse Europe Foundation was launched at the gameslab of Humboldt University in Berlin. The initiative aims to sustainably integrate immersive technologies and artificial intelligence in the education sector in order to create personalised and inclusive learning spaces and thus promote equal digital opportunities. […]
