Abstract

1.    Introduction

In the last couple of years, artificial intelligence (AI) revolutionized several fields, especially natural language processing (NLP), computer vision, and text generation. Traditional fixed-rule-based systems were substituted, to a large extent, with deep learning models capable of learning from data directly. One of the first breakthroughs in this shift was the 2017 release by Vaswani et al. of Attention is All You Need, in which they introduced the transformer architecture [1]. Not only did the transformer model offer a new technological fix, but it also radically altered how AI systems learned. Before the transformer model came into the picture, Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks were the most widely used techniques employed for sequential data processing [2, 3]. These models processed input sequentially, based on the output of the previous step, a property that barred parallelization and limited their ability to model long-range dependencies. From a computational perspective, these models suffered from O(n²) complexity due to their sequential nature. In contrast, the transformer’s self-attention mechanism allows the model to focus on all array elements (tokens) simultaneously, enabling parallel computation and ideally reducing the time complexity to O(1) [1, 4].

The success of the transformer was not entirely due to self-attention. There were several complementary pieces of the architecture, such as multi-head attention, layer normalization, residual connections, positional encoding, and encoder-decoder design, that have essential roles to fulfilled in enabling faster learning and more consistent output [1]. These all needed to be grasped to appreciate the overall operation of the transformer model.

This architectural innovation did not remain at the level of theoretical exploration; it has inspired the development of numerous practical models. Outstanding examples are BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), and DALL-E. BERT significantly improved contextual linguistic understanding, the GPT family has shown strong talent in text generation, and DALL-E set a new benchmark in multimodal AI by converting input text to images [5, 6].

Such technologies were not only limited to scholarly research; they were integrated into everyday tools. Voice assistants, content creation tools, language translation tools, and chatbots extensively employed transformer-based technology. Text and image data were processed in parallel as exemplified by ChatGPT and Meta's LLaMA. Similar examples were T5 models created in the form of a text-to-text configuration that make their way to being widely utilized in machine translation and summarizing texts [7]. Similarly, models focused on audio processing such as wav2vec 2.0 and SpeechT5 were used in natural language recognition and production processes in voice assistants and speech-based interaction systems [8, 9]. In this field, the Whisper model developed by OpenAI stood out in multilingual speech recognition and transcription tasks, while the Sora model offered by the same organization represented the latest point in multimodal artificial intelligence applications with its ability to produce realistic videos from text input [61, 63].

Transformer systems have, besides BERT, GPT, and DALL-E, spread far wider as well. More complex architectures have emerged in recent years, including multimodal models like GPT-4 and GPT-4o, long-context models like Claude 3 and Gemini, and open-source, efficiency-driven models like LLaMA 3 and Mistral [10-13].

This study aimed to provide an extensive review of the transformer architecture that was at the core of current generative AI. It provided its technical foundation, timeline, and actual usage. The article began discussing the architectural advantage of transformers as opposed to established methods like RNN and LSTM. It continued by analyzing the primary components of the architecture: multi-head attention, layer normalization, positional encoding, residual connections, and the encoder-decoder framework. Later, the study analyzed the patterns of use and design of well-known models such as BERT, GPT, DALL-E, GPT-4, GPT-4o, Claude 3, Gemini, LLaMA 3, Mistral, Sora, and Whisper. Finally, it was engaged in a critical discussion on the social and ethical dangers of generative AI, that is, hallucination generation, amplification of prejudice, copyright, and energy consumption with large models.

Consequently, the structure of the paper is as follows: Section 2 is a review of literature, Section 3 methodology description, Section 4 results of chosen model analyses, and the final section is a general conclusion and evaluation.

2.    Methodology

The present research used a qualitative research approach to comprehensively examine the technical architecture, learning habits, usage scenarios, and ethical aspects of transformer-based generative AI technologies. The methodology framework for the present work was informed by a comparative content analysis strategy based on current literature.

In the first step of the research, three simple models were selected: BERT, GPT, and DALL-E. But with the general development of next-generation systems based on the transformer model, the research scope was extended. The research encompassed contemporary models such as GPT-4, GPT-4o, Claude 3, Gemini, LLaMA 3, Mistral, Sora and Whisper. The models not only varied technically but also in the application domains they addressed, their capacity, multimodal input capability, and contextual processing capability.

All the models were checked on four major parameters:

·       Learning structure: How the model was trained, what pre-training techniques were used,

·       Contextual ability: How meaning was constructed between textual and visual elements,

·       Mode of production: How outputs were produced (predictive, masked, multimodal),

·       Relationship with ethical-social impacts: Sensitivity to issues such as prejudices, information security, energy use.

The data sources utilized for the analysis encompassed academic publications by the model developers, technical reports, open source documents, and independent academic studies on these systems.

This methodological approach has facilitated a more profound comprehension of the models' functionality at the architectural level, as well as their daily life counterparts, their impact on users, and their societal implications. Therefore, the objective was not merely to provide a technical description of the systems, but rather to engage in a multifaceted examination of their functionality, perception, and associated risks.

3.    Literature Review

3.1. Pre-Transformer Techniques: Recurrent Neural Networks and Long Short-Term Memory

Handling sequential data such as the sequence of words in sentences has been a focal point for artificial intelligence research, particularly in the context of natural language processing (NLP). With every step in a sequential set of data, the information from the previous steps needed to be accounted for, thereby rendering a basic artificial neural network inappropriate for handling such data. In order to solve these issues, recurrent architectures emerged and, as a result, models such as RNNs and subsequently LSTM networks were created. Both approaches had significant shortcomings that emphasized the need for more efficient and robust architectures [14]. Transformer architecture filled this gap. This section provides a detailed analysis of the natural properties of pre-transformer approaches and explains why they are not optimal.

3.1.1. RNN Architecture and Most Important Features

Dealing with sequential data was one of the problems of artificial intelligence. RNN, as a solution to these kinds of problems - i.e., maintaining a memory of past data at every time step - became one of the earliest successful architectures for modeling relationships between sequential data. In the first recurrent neural network model proposed by Elman (1990), the model was not solely reliant on the data present at any time step but also on the recollection of the information at the last step in processing each new example [2]. Consequently, this architecture enabled the model to learn the temporal sequence of the data and establish useful relationships among neighboring items.

Through this process repeated over and over again, several times, RNNs were able to "remember" what had been experienced previously and make decisions at the present time step based on these [15]. For example, an RNN being fed a sentence might potentially look at words that have already been encountered in the sequence so that it could better consider syntactic structure and semantic content.

RNN architectures had some limitations. Firstly, because the data was fed sequentially, it would not have been possible to execute operations in parallel during training. This increased the training time when working with large datasets. Secondly, RNNs faced problems such as vanishing gradient or exploding gradient when working with long sequences [14]. These problems made it difficult for the model to remember information from the distant past. These problems were demonstrated in various studies in the literature: Cho et al. [16] emphasized that RNN-based models had difficulty dealing with long sentences; Bahdanau et al. [17] proposed the attention mechanism to alleviate these limitations; Sutskever et al. [18] showed that the sequential processing requirement limited the training speed of the model.

For these reasons, RNN-based approaches were successful in some tasks, but they had limitations in terms of preserving information on long sequences, performing fast training, and working efficiently. Because of these problems, it became necessary to develop more robust models, especially more advanced network structures such as LSTM.

 

 

3.1.2. Limitations of RNN Architectures

Although RNN initially provided significant improvements when working with sequential data, they also introduced some problems. For example, gradient vanishing and gradient exploding [14]. During training, as the error signal went back in time, the gradients sometimes became so small that learning almost stopped. Sometimes the opposite happened, the gradients became very large, which disrupted the balance of the model. These problems became even more pronounced, especially in long sequences, so the model had a hard time learning information from the distant past.

Another problem was that RNNs processed data sequentially. In other words, only one data was processed at each step, meaning it could not perform parallel computations. When working with large data sets, the training stages took a long time [18]. All these limitations led to RNNs not being able to work efficiently enough in larger structures and eventually led to the development of more powerful architectures.

3.1.3. Development of LSTM Structure

RNN were not always successful in learning long-term dependencies. For this reason, the LSTM architecture was developed by Hochreiter and Schmidhuber [3]. LSTM was able to better control the flow of information by adding some special “gate” mechanisms to the classical RNN structure (Figure 1). In this structure, components such as input gate, forget gate and output gate decided which information to keep in the cell state, which to update and which to delete. Thus, the model could preserve important information from the past for a long time, while it eliminated unnecessary information over time.

Figure 1. LSTM Cell Structure

This structure greatly reduced the gradient loss problem, especially when working with long sequences, by ensuring that necessary information was preserved and unnecessary information was forgotten. Thus, LSTM achieved great success in sequential tasks such as machine translation, speech recognition and text generation [20].

However, LSTMs remained limited in parallel computation and continued to have problems in terms of training time and resource consumption, especially for very long sequences. For these reasons, one of the main goals in later developed alternative architectures was to eliminate the sequential processing dependency and to be able to model long contexts directly [1].

3.1.4. Common Structural Limitations of RNN and LSTM

Although the LSTM architecture significantly reduced the weaknesses of classical RNNs in modeling long-term dependencies, both structures did not overcome some fundamental limitations. In particular, the step-by-step progress of the training processes increased both the time cost and limited the efficient use of hardware resources when working on large data sets [3, 21].

Table 1 was presented to see the fundamental limitations of these two architectures more clearly:

Table 1. Fundamental Structural Features and Limitations of RNN and LSTM Architectures

Feature	RNN	LSTM
Memory Capability	Short-term	Can retain long-term information
Gradient Problem	Pronounced	Largely mitigated
Parallel Processing	Not possible	Not possible
Performance on Long Sequences	Poor	Improved but still limited
Computational Efficiency	Low	Relatively better but still limited

While both architectures were theoretically capable of handling long sequences, they demonstrated challenges in transferring information beyond a specific context length in practical applications. This phenomenon was particularly pronounced in the context of extensive texts or data characterized by intricate relationships, where the efficacy of prior knowledge was shown to diminish over time, thereby impeding the performance of the model. Khandelwal et al. (2018) posited that the primary reason for the encountered issues with these models was attributed to their static memory structure [22]. In a similar vein, Lake and Baroni demonstrated in their experiments that models such as RNN and LSTM were not as proficient as humans in abstract thinking and generalization [23].

Although models such as RNN and LSTM were an important step for sequential data, they began to give way to more advanced models due to some limitations.

3.1.5. The Emergence of the Need for Transformer Architectures

The constraints on the structure of RNN and LSTM drove researchers to integrate new architectures that had the capability to model more effective and long-term relationships directly. The Transformer architecture evolved on this path and greatly influenced sequential data processing. The incorporation of the self-attention mechanism enabled the Transformer to represent the relations among all input data simultaneously, hence enabling the learning of long-range dependencies and parallel data processing [1]. Specifically, the parallel processing capability offered by the self-attention framework served to accelerate and expedite training of the Transformer in handling large datasets [1]. Thus, computational resources were used more effectively. Besides that, with its ability to establish direct connections between distant terms, the Transformer was able to produce accurate results even in large data decays [24]. The Transformer model, thanks to its strong abilities, was applied in many areas beyond the natural language processing field, including image recognition [25], speech modeling, and multimodal learning. Hence, the Transformer broke the intrinsic limitations shared by all the previous iterative methods, paving the way for the innovation of a new generation of models that formed the foundation of the majority of current artificial intelligence applications.

3.2. The History and Evolution of Transformer Architectures

The development of transformer-based models began in 2017 with the introduction of the Transformer architecture, which implemented a self-attention mechanism that allowed for parallel processing. In 2018, this innovation led to two major advancements: BERT, which introduced bidirectional context and masked language modeling, and GPT-1, which initiated autoregressive language modeling. GPT-2 followed in 2019, showcasing improvements in large-scale language generation. GPT-3 launched in 2020 with 175 billion parameters and introduced few-shot learning. In 2021, DALL-E enabled text-to-image generation using CLIP integration. By 2022, Whisper emerged, offering multilingual speech recognition using weakly supervised learning. The year 2023 saw the rise of GPT-4 with multimodal capabilities and long context windows, and the release of open-source, resource-efficient Mistral/Mixtral models based on Mixture of Experts architecture. In 2024, Claude 3 introduced constitutional AI and long-text coherence, LLaMA 3 emphasized high efficiency, and GPT-4o enabled real-time multimodal processing. Additionally, Gemini 1.5 Pro was developed by Google DeepMind with a security focus, and Sora introduced text-to-video generation. These milestones, illustrated in Figure 2, reflect a rapid and diverse evolution of transformer models across modalities and use cases.

Figure 2. Evolution of natural language processing models based on Transformer architecture, highlighting key milestones from 2017 to 2024.

3.2.1 What is a Transformer?

The 2017 paper "Attention is All You Need" by Vaswani et al. was widely considered to mark a turning point in artificial intelligence, particularly in the field of natural language processing [1]. Figure 3 shows the original structure of the transformer architecture given in the paper by Vaswani et al.

The Transformer model eliminated the need for sequential processing by offering an architecture based on a self-attention mechanism. Instead of processing data sequentially, the model had the capacity to handle all components of the input simultaneously. This enabled it to capture long-range dependencies more effectively and significantly accelerated the training process. In particular, in applications involving large datasets, the Transformer model demonstrated a clear advantage over previous models in terms of both speed and efficiency [17].

 

Figure 3. The Original Structure of the Transformer Architecture. Vaswani et al. (2017) [1].

3.2.2 Structure and Operation of Self-Attention Mechanism

The self-attention mechanism, which forms the basis of the Transformer architecture, allows each element in a sequence to establish relationships with all other elements simultaneously [1]. This allows information to be processed simultaneously rather than sequentially, enabling more accurate learning of long-distance relationships [24].

The self-attention mechanism started by converting each input element to three vectors i.e., query, key, and value. Vectors were created by using specific weights on the input embeddings [1]. Each query vector was then compared to all the key vectors. Attention scores were found, and the relations between these elements were determined. The scores were normalized to the size of the key vector and were weighted by the softmax function [1]. The weights were multiplied by the value vectors to produce context-sensitive output [1].

One definite advantage of this mechanism was that it could provide direct access between elements, even in lengthy sequences [24].

Traditional models like RNN and LSTM had limitations in how effectively they could retrieve information from far away in the past. However, self-attention could handle relationships at any distance with the same ability [1], [14]. Furthermore, carrying out operations in parallel greatly decreased training times and helped speed up the modeling of large data sets [26].

Figure 4. Self-Attention Mechanism [1].

The flowchart given in Figure 4 illustrates the main steps of the self-attention mechanism in Transformer models.

3.2.3 Transformer Architecture: Technical Components

Transformer architecture was based on a basic principle such as self-attention, but it also consisted of various auxiliary components for this mechanism to work effectively.

3.2.3.1 Multi-Head Attention

In the Transformer architecture, the self-attention mechanism tried to understand the relationship of each word (token) with all other words in the sequence. However, doing this relationship through a single attention mechanism (single attention head) might have caused complex contextual structures not to be captured well enough. For this reason, the multi-head attention structure was preferred in Transformer [1]. Thanks to this structure, the model received and processed information from multiple attention heads at the same time. Thus, each head could evaluate the relationships between the elements in the sequence from different perspectives; for example, while one head learned grammatical relations, another can capture the semantic context [1].

Each head performed calculations using its own special query, key and value matrices. The output of these attention heads was combined and then passed through a linear transformation and provided to the model [1]. Transformer thus generated a richer and more multi-dimensional representation by comprehending the context in different ways. One of the largest advantages of this architecture was that it could comprehend the relationships between words better, especially in a large sequence [1].

3.2.3.2 Positional Encoding

Given the Transformer architecture's inherent simultaneous processing of words, it lacked an inherent understanding of word order. However, in sequential structures such as language, the position of the word directly affected the meaning it carried. Consequently, positional encoding was employed to furnish the model with order information [1]. Thus, by learning the order information together with the data, the model learned not only the meaning of the words but also their place in the sentence.

3.2.3.3 Layer Normalization

As in deep neural networks, some normalization techniques were needed in Transformer architecture to ensure that the training process was fast. At this point, layer normalization applied to the output of each sublayer helped the model maintain balance during the learning process [27].

Layer normalization scaled the neuron activations in each layer according to the mean and standard deviation values [27]. This process ensures that the data was kept within a certain distribution range, thus preventing negative effects such as gradients growing too large or disappearing.

Layer normalization in Transformer was usually applied together with residual connections. This increased training efficiency, especially in multi-layer structures, and made the model more stable [1].

3.2.3.4 Residual Connections

There are residual connections employed between layers in the Transformer model to ensure information flow. The model was designed in such a way that the output from a layer can be added directly to the input it received [28].

This approach guaranteed that the output from each lower layer was transmitted to the next layers without distortion. Incidentally, it also guaranteed a more effective passage of Declivities during back propagation. This design was mathematically usually depicted as follows:

Output = LayerNorm(x + Sublayer(x))

Where x was the input data to the layer, and Sublayer(x) was the output of the respective attention or feed-forward layer. Because of this structure, information was preserved more clearly and the learning was more stable [28].

3.2.4 Information Flow in Encoder-Decoder Structure

Two elements of information flow in the Transformer model were encoder and decoder [1]. The encoder side converted input information (e.g., words within a sentence) and outputted a contextual representation of each unit. Here, self-attention occurred. Each word updated its own information by looking at how it related to other words. As a result, the encoder was enriched with information of each member of the array on the basis of its data as well as its relationship with other members of the entire array [1]. The decoder produced a new output using this information from the encoder. For example, when translating an English sentence into French, the decoder focused on both the words it produced (masked self-attention) and the contextual information provided by the encoder (encoder-decoder attention) while creating each word step by step [1]. This bidirectional attention system ensured that the output was both grammatically correct and semantically consistent [17].

This interaction between encoder and decoder strengthened the flow of information, especially in sequential data processing problems. The model did not only depend on the direct order of the input data; it also produced output by evaluating the context from a broader perspective. This structure offered a much more flexible and effective learning process compared to traditional sequential models [1, 17].

3.2.5 Comparison of Transformer with RNN and LSTM Architectures

The Transformer architecture developed a sequential data processing method that is very different from previous methods. While traditional models process data sequentially, transformers process all input elements simultaneously thanks to their self-attention mechanism [1]. This allows each element to establish a direct connection with all other elements, enabling it to learn the context much more quickly and effectively. Transformers learn context much more quickly and effectively by processing all input elements simultaneously thanks to their self-attention mechanism.

This approach allowed the model to establish strong relationships between distant locations even in long sequences. For example, while the relationship between a word at the beginning of a sentence and a word at the end could be established in a few steps in Transformer, learning this relationship became much more difficult in models such as RNN and LSTM [14, 3].

In order to reveal the differences between Transformer architecture and traditional models such as RNN and LSTM, a comparison of the three structures mentioned was presented in Table 2.

Table 2. Comparison of RNN, LSTM, and Transformer Architectures

Feature	RNN	LSTM	Transformer
Data Processing Method	Sequential (step-by-step processing)	Sequential (step-by-step processing)	Parallel (simultaneous processing of entire input)
Ability to Model Long-Range Dependencies	Weak (due to vanishing gradient issue)	Moderate (enhanced via gating mechanisms)	Strong (via direct self-attention-based modeling)
Parallelization Capability	Not supported	Not supported	Fully supported
Training Complexity	High (O(n))	High (O(n))	Lower (O(1) per step under idealized assumptions)
Computational Efficiency	Low	Relatively low	High
Contextual Learning Capacity	Limited (short-range dependencies only)	Improved	Excellent (captures global dependencies effectively)
Memory Consumption	Low	Moderate	High

Transformer had significant advantages over previous methods, not only with its ability to perform parallel processing, but also with its ability to model long-distance relationships more effectively and to provide a significant speed advantage in the training process.

3.3 Transformer-Based Models: BERT, GPT, DALL-E and Current Applications

Transformer architecture attracted attention not only with its theoretical innovations but also with its practical applications. In this section, how BERT, GPT and DALL-E, which were effective models based on Transformer architecture, worked, how they were trained and in which tasks they were used, will be comparatively discussed. In addition, multimodal systems such as GPT-4 and GPT-4o and open-source models such as Claude 3, Gemini, LLaMA 3, Mistral, Sora and Whisper that could process long contexts were evaluated within the framework of their architectural structures and application scenarios.

3.3.1 BERT: Transformer-Based Semantic Learning

BERT (Bidirectional Encoder Representations from Transformers) model was developed by Devlin and colleagues in 2018 [29]. Most models before BERT processed text in one direction, either from left to right or from right to left. BERT, on the other hand, was able to capture the true meaning of a word in context more accurately by considering both the previous and next words simultaneously [29].

The preliminary training of the model was carried out through two basic tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). Thanks to these tasks, BERT was able to learn semantic relationships at both word and sentence levels [29]. Experiments have shown that the model has high success in many NLP tasks such as sentiment analysis, text classification, and question-answer systems.

BERT's success in a short time has paved the way for the development of different variants. For example, the RoBERTa model increased performance with changes such as longer training, larger datasets, and abandoning the NSP task [30]. Distillbert reduced the model size and made it work faster [31]. The ALBERT model, on the other hand, emphasized efficiency with techniques such as parameter sharing and embedding factorization [32].

BERT's success was concreted when it achieved 93.2% F1 score in Stanford's SQuAD 1.1 question-answer test, providing results close to human level [29]. Additionally, Devlin et al. (2019) also showed that BERT could function at high levels of accuracy even in low-density data environments and could seamlessly switch to transfer learning [33].

Extensive studies conducted in the literature revealed BERT learned grammatical knowledge well but was deficient in more complex tasks such as multi-step reasoning and logical inference [34]. Sun and others showed task-based strategies of fine-tuning were able to improve performance on such tasks [35].

Despite all these advancements, BERT did have some drawbacks, such as an enormous model size, immense processing power requirements, and the inability of it to see the full context of the long texts. Because of that, there was still a requirement to develop more efficient and more potent models. In this aspect, BERT is not only a powerful language model itself, but also a critical breakthrough that paved the path for many other models to follow.

3.3.2 GPT: Autoregressive Models and Creative Language Generation

The Generative Pre-Trained Transformer (GPT) series was developed by OpenAI and had a great impact in the field of language generation. Unlike BERT, GPT adopted an autoregressive approach; that is, it processed the text in order from left to right and tried to predict the next word at each step. This structure provided a great advantage in natural and fluent text production [36].

The GPT model used only the decoder part of the Transformer structure. Thanks to the causal self-attention mechanism, this model made word predictions by hiding the words that had not yet come in the sentence. In other words, when predicting the next word, it used only the words it had seen before. Thanks to this method, the model constructed sentences in order and determined each new word according to the previous words. This system was used especially in areas such as chatbots or story writing [37].

The first version, GPT-1, was introduced in 2018 and showed that it could be fine-tuned for various tasks after being pre-trained on a large amount of general text data. GPT-2, released in 2019, proved that it could produce texts that were very close to those produced by humans, with a larger model capacity and more comprehensive data sets [38].

GPT-3, introduced in 2020, became the largest language model of that period with 175 billion parameters. GPT-3 was able to successfully perform many tasks with only a few examples (few-shots), and sometimes even without examples (zero-shots) [37].

GPT-4, introduced in 2023, was another model with multimodal capabilities. In other words, it could work on both text and visual inputs. With GPT-4, the model's accuracy, reliability and performance on complex tasks were further increased.

GPT-based models, such as ChatGPT (text generation and chat systems), GitHub Copilot (code generation and completion), and Notion AI (document management and content generation) represented different examples of GPT use cases.

However, GPT models also had limitations. Especially in terms of producing correct information, "hallucination" (unreal information production) was sometimes experienced. In addition, they had difficulty following the subject in very long texts. The large size and high training costs of these models led to significant difficulties in both development and implementation processes [39, 40].

3.3.3 DALL-E: Transition from Text to Image

DALL-E was a multimodal model developed by OpenAI that could produce original images from written texts. Based on the Transformer architecture, DALL-E transformed given text descriptions into visual content. In this way, it was seen that creative content could be produced not only from language but also from visual data [41].

DALL-E 1, introduced in 2021, was an important first step in producing meaningful images from text descriptions. Then, DALL-E 2 not only produced higher-resolution results; it also allowed users to make creative edits to images with features such as “inpainting” [42].

DALL-E 3, on the other hand, made it possible to match entered texts with more consistent, detailed and artistic images with more advanced conceptual matching capabilities.

The CLIP model was responsible for the success of DALL-E. CLIP operated in a manner where it aligned images and texts in the same mathematical space. This allowed DALL-E to learn from written texts and create appropriate images. CLIP acted as a bridge between text and images upon which DALL-E produced more accurate and text-appropriate images [42, 43].

DALL-E had a wide range of applications. It was used in sectors such as art, advertising, game design, and media content production. However, it also brought about important ethical questions such as copyright ownership and ethical usage limits for images produced with artificial intelligence [44]. The situation brought to the agenda of policy makers and technology developers’ new roles and regulation requirements [45].

Table 3 shows the basic strengths and weaknesses of the BERT, GPT, and DALL-E models comparatively.

Table 3. Comparative Overview of the Core Strengths and Limitations of BERT, GPT, and DALL-E Models

Model	Strength	Limitation
BERT	Strong in contextual analysis	Limited in creative generation
GPT	Fluent and natural text generation	Risk of hallucination
DALL-E	Powerful in visual and creative	Issues of copyright and ethics

3.3.4 GPT-4 and GPT-4o

GPT-4 was one of the first multimodal models OpenAI developed in 2023 that was capable of processing image data and text data at the same time. GPT-4 was also been able to effectively carry out not only text-related tasks but tasks related to table analysis, visual content recognition and understanding graphs. One of the most important characteristics that distinguish this model from the previous models was the context window which was up to 128,000 tokens. This characteristic enabled it to produce coherent and complete sentences even on very long texts [46, 47].

Figure 5. Comparison of GPT-4, GPT-4 (no vision), and GPT-3.5 on standardized exams, reported by OpenAI GPT-4 Technical Report. [46]

GPT-4 achieved spectacular success in high-stakes multi-step reasoning assessments such as law, engineering, and university exams. For instance, it scored in the top 10 percentile of the US Bar Exam, 93rd percentile in SAT Evidence-Based Reading & Writing, 89th percentile in SAT Math, and 99th percentile in GRE Verbal. These signs confirmed its human-level performance in language and quantitative domains and its superior reasoning and problem-solving capabilities. Figure 5 illustrated a graphical overview of the above results relative to GPT-3.5.

With such success, OpenAI introduced the GPT-4o (omni) model in 2024. As a multimodal model, GPT-4o did not only interact with text and images, but even accepted speech input. Such an architecture, which was capable of hearing speech in real time, responding, and emulating human-specific modes of expression like intonation, made human-computer interaction more natural and faster [47].

One of the most striking features of GPT-4o was that it can provide similar quality responses with lower computational power. This system, which could operate with much less energy consumption compared to GPT-4, allowed artificial intelligence to be accessible even on mobile devices and low-equipped computers. In this respect, GPT-4o made generative artificial intelligence more economical, faster, and widely available [47].

These advanced features enabled GPT-4 and GPT-4o to be used not only for writing; but also, in many areas such as voice assistants, image analysis programs, data processing applications, and educational tools. However, these systems can still sometimes produce hallucination, might not fully reveal their sources, and could cause excessive energy consumption while operating [39]. Therefore, both the strengths and risks of these models needed to be carefully evaluated.

3.3.5 Claude 3: Consistency and Semantic Depth in Long Contexts

The Claude 3 model developed by Anthropic stood out due to its ability to analyze long texts and maintain semantic integrity among these texts [10]. Although the infrastructure of this model was based on the Transformer architecture, thanks to some technical improvements made in the attention mechanism, both memory usage was made more efficient and it was ensured to work by preserving the information in long texts better [48].

Compared to other models, Claude's distinguishing feature was its ability to work with a context window exceeding 200,000 tokens [10]. In other words, Claude could read multiple documents simultaneously, summarize long texts and give more consistent answers to questions that required progressive thinking. This advantage of Claude made a significant difference especially in long and detailed content such as academic papers, legal documents or technical reports [10].

During the training of the Claude 3 model, the "Constitutional AI" approach was combined with the systematic use of human feedback (RLHF) to ensure that certain ethical principles were preserved in the model's outputs [49]. With this approach, the model was first limited according to certain principles, and then fine-tuned accordingly to these limitations. Thus, Claude 3 focused on producing reliable and responsible content, especially in sensitive areas such as law, health and education.

However, Claude 3 also had some limitations. In particular, the response filtering mechanism developed within the framework of ethical rules might have caused the model to avoid providing information about some sensitive issues or to produce extremely limited responses. This situation was criticized by some users as being too restrictive [50].

As a result, Claude 3 was a powerful Transformer model that stood out in maintaining consistency in long texts, maintaining semantic integrity and producing content in accordance with ethical principles. Although it did not have the ability to process multimedia (multimodal) data, it offered a strong alternative in terms of reliability and contextual accuracy in text-oriented applications.

3.3.6 LLaMA 3

The LLaMA 3 models developed by Meta were one of the language models that made an impact especially in the open source world. This model, which was offered in two different sizes with 8 billion and 70 billion parameters, was very powerful but was popular among developers due to its free accessibility [51]. These models, which were shared on Hugging Face and similar platforms, were frequently preferred for both academic studies and special application development. The architecture of LLaMA 3 was restructured to use the Transformer structure more efficiently with advanced token location coding systems and more optimized attention layers. The data sets used in its training were also diversified, allowing the model to respond better to multilingual texts [52]. With these features, it became functional in a wide range of areas from translation applications to dialogue systems, from classification tasks to creative content production.

3.3.7 Gemini: Google's Multimodal Approach

In 2023, Google DeepMind unveiled Gemini, an advanced multimodal artificial intelligence system. This system was capable of processing diverse data types, including text, images, audio, and code, in a unified manner [11]. The device has the capacity to process information in various forms, including text, images, audio, and code, concurrently. It could process information such as text, image, audio and code at the same time. In this respect, it was particularly competitive with other multimodal models such as GPT-4o.

Gemini's training was done with a very large amount of text, image, audio content and code data. In this way, it could perform many different tasks such as writing text, generating code, making visual explanations, recognizing speech and summarizing content. Google's Bard chat system also used the infrastructure of this model.

One of the most important features of the model is that it focused on providing safe and accurate results. Various security filters were used during its training to reduce the production of false information, prevent bias and protect the user [11]. In this respect, it stood out among the new generation artificial intelligence models that prioritized user security.

In addition, Gemini had versions that can work on different devices. Lightweight models such as the “Gemini Nano” were designed specifically for use on phones and low-power computers. In this respect, it was suitable not only for powerful servers but also for everyday devices [11].

As a result, Gemini combined what Google had learned from its previous models to offer a more powerful, secure and flexible artificial intelligence model. It was both technically advanced and suitable for use in different areas.

3.3.8 Mistral and Mixtral

Mistral and its specialized version Mixtral, developed by France-based Mistral AI, were open source language models that exhibited very strong performance despite their small size. Mistral 7B could produce results at the GPT-3.5 level despite having only 7 billion parameters [53].

The Mixtral model worked with the "Mixture of Experts" (MoE) architecture. In this structure, only a few expert submodels (experts) were activated for each input. This architecture worked as follows: there were many "expert" submodels in the model, and for each input (for example, a piece of text), the system activated only a few experts that were most suitable for that input, not all experts. Thanks to this architecture, a successful balance was established between high accuracy and low resource consumption. Mixtral was widely used in tasks such as content generation, code completion, and document summarization [53].

Since both models were published as open source, developers could customize these systems according to their own projects. This open source advantage allowed developers to examine the internal structure of the models, modify them according to their own needs, retrain them for specific applications, and use them freely in both academic and commercial projects.

3.3.9 Whisper: Speech-to-Text Transformation

Whisper was an open speech-to-text model developed by OpenAI. It was based on the Transformer architecture and was trained on 680,000 hours of multilingual and accented speech data of various origins [61, 62]. Whisper offered a world-wide solution, not merely English but over 90 languages supported.

One of the outstanding features of Whisper was that it provided successful results even with low-quality audio recordings and environments with high background noise. This increased the value of the model, especially in applications such as subtitle production, call center analytics, podcast transcriptions and accessibility services. In addition, successful results were obtained in tasks such as direct speech translation.

The technical architecture of the model, the methods and results of training on large-scale weakly supervised data were described in detail in the technical report presented by Radford et al. [62]. This work showed that Whisper was not only a transcription model, but also a multilingual and multitasking audio processing system.

However, it was noted that it has limitations such as not being able to fully preserve the context in conversations in some languages and incorrectly identifying proper nouns. Thanks to the open-source nature of the model, it was possible for developers to integrate and adapt it to different applications.

3.3.10 Sora: Transition from Text to Video

Announced by OpenAI in 2024, Sora was an artificial intelligence model that could produce video output from text input [63]. This model had the capacity to transform a few-sentence scenario into an audiovisual video in seconds. Sora drew attention with its visual continuity, physical consistency, and naturalness of scene transitions in video production.

Sora combined a number of artificial intelligence techniques: It combined Transformer-based language modeling and Diffusion-based image modeling. The hybrid model allowed the model to provide narrative coherence as well as visual realism. In the introduction of the model, complex interaction character scenes, spatial changes, and detailed simulations of physical events were successfully produced. However, this powerful production capacity also brought with its serious ethical discussions. In particular, the production of fake videos similar to deepfake technologies posed risks in areas such as information security, personal rights, and content manipulation. Therefore, it was emphasized that content marking, watermarking and verification protocols should be developed for the safe and ethical use of Sora.

3.4 Ethical, Social, and Environmental Discussions

Transformer architecture created great advances in artificial intelligence by providing high efficiency. However, these achievements also brought with them some ethical, social and environmental problems. This section examined the main ethical problems posed by transformer-based systems.

3.4.1. Information Security and Misleading Content

Transformer-based language models, especially in systems such as GPT, which were large-scale generative structures, sometimes faced the problem of producing false information, called "hallucination". This meant that the model produced unrealistic but convincing statements. For example, the model might have referred to a scientific article or person that did not exist in reality. Such outputs carry serious risks, especially in areas where information security is of vital importance, such as education, health and law [54]. Ji et al. have systematically examined the tendency of large language models to produce false information and revealed that such errors were caused by the learning strategies of the model [54].

Similarly, Maynez et al. [55] stated in their studies on summarization systems that the model outputs might not have matched the source text and this had the potential to mislead the user. The phenomenon of hallucinations not only leads to the production of false information; it can also lead to damage to user trust and the spread of disinformation. In this context, the circulation of false information, especially on social media platforms, could lead to trust problems at the social level [56]. Therefore, it was of great importance to approach the information generated by large language models cautiously and to support these systems with security filters.

In addition to text-based models, voice-enabled models such as Whisper were shown to pose serious concerns. Such models could potentially have transcribed intimate or personal speech without consent, raising new data privacy and surveillance ethics concerns. As more pervasive speech recognition technology was developed, concerns around their ethical deployment and responsible data processing practices became increasingly salient.

3.4.2. Biases and Discrimination

Given that transformer-based systems were trained on extensive text or visual datasets, there is a tendency for these systems to learn and reinforce social biases inherent in these data. Representational biases in categories such as gender, ethnicity, religion, and disability can be reproduced in model outputs. This phenomenon can result in digital inequalities, algorithmic discrimination, and the exclusion of specific user groups. To illustrate, word embedding models have demonstrated that the term "woman" is associated with domestic tasks, while "man" is associated with technology. Such biases have the potential to contribute to the propagation of sexist stereotypes in model outputs [56].

In addition, lack of data openness and transparency also caused these biases to become even more invisible. Bender and Friedman emphasized that developers may not be aware of potential biases in their data sources because many language models were not explicitly stated on which data they were trained [57]. This paved the way for the model to perpetuate discriminatory stereotypes independently of the developers. It is very difficult to universally detect and eliminate these prejudices, especially in multilingual or multicultural systems.

3.4.3. Copyright and Content Ownership

Generative AI systems have also raised important ethical debates in terms of copyright violations. In particular, datasets used in training models that can produce visual content, such as DALL-E, often contained copyrighted images. In their study titled "Excavating AI", Crawford and Paglen revealed that a large portion of popular image datasets were collected from platforms such as Flickr without user permission, and that this posed serious risks to both visual property rights and privacy. Models trained on these datasets reproduced content without user permission, creating serious uncertainties about content ownership [58].

This issue is not exclusive to static images. Text-to-video models, such as OpenAI's Sora, expand the domain of copyright and ethical concerns to the realm of generated videos. These systems possess the capability to synthesize video content that mimics existing copyrighted material or the likeness of individuals, thereby giving rise to legal and ethical ambiguities related to deepfake creation, ownership rights, and unauthorized use of identity or branding.

In addition, it is still unclear who owns the content produced by AI – the artist or the company that developed the algorithm. McCormack et al. [44] emphasize that there were serious gaps in how works of art produced by AI should be defined in current legal systems.

3.4.4. Explainability, Energy Consumption and Sustainability

Transformer-based large language models are typically characterized as "black boxes," a term utilized to explain the challenge of understanding the decision-making process and the reasoning behind the conclusions drawn by the model, encountered by both users and developers. The issue of transparency could lead to the concern of not being trusted, especially in sensitive fields such as medicine, judicial proceedings, and education. It is difficult to measure the model's responses' accuracy or objectivity unless the data or axioms upon which the responses are based are specified beforehand. Doshi-Velez and Kim argued that explainable artificial intelligence is needed not just for user understandability but also to enable developers and regulatory agencies to control systems with greater transparency [59]. Meanwhile, the training and utilization of massive language models have had the potential to make a substantial effect on the environment.

Strubell et al.'s work also shows that the training of a model like BERT caused greenhouse gas emissions that equated to several thousand kilograms of carbon dioxide [60]. The volume measured was significantly larger than the carbon footprint generated by a person driving a car for an extended duration or heating their home. Thus, current research in artificial intelligence is mostly aimed at building energy-efficient models. Research is conducted to create more effective and greener systems through techniques such as model compression and knowledge distillation.

But the "green artificial intelligence" (Green AI) paradigm, which focuses on creating systems with a priority on transparency and ethical considerations, is increasingly in the spotlight. The Green AI framework tries to develop models that are characterized by low energy consumption and better interpretability and accountability. For example, it matters that a model does not generate gender or racial bias at the expense of energy efficiency. In general, contemporary artificial intelligence research transcends the sole focus on technical progress; it is anticipated that such machines also adhere to ethical guidelines, demonstrate ecological sustainability, and respect societal norms.

Table 4. Deployment and infrastructure specifications of large language models (LLMs), highlighting key environmental impact factors such as critical power requirements, energy efficiency (PUE), water usage efficiency (WUE), and carbon intensity (CIF). These infrastructure attributes play a central role in determining the carbon footprint generated during training and inference [64]

Model	Critical Power (kW)	PUE	WUE (on-site, L/kWh)	WUE (off-site, L/kWh)	CIF (kgCO2e/kWh)
GPT-4.1	10.20	1.12	0.30	3.142	0.3528
GPT-4.1 mini	10.20	1.12	0.30	3.142	0.3528
o4-mini (high)	10.20	1.12	0.30	3.142	0.3528
GPT-4.5	10.20	1.12	0.30	3.142	0.3528
o3	10.20	1.12	0.30	3.142	0.3528
o3-mini (high)	10.20	1.12	0.30	3.142	0.3528
o3-mini	10.20	1.12	0.30	3.142	0.3528
o1	10.20	1.12	0.30	3.142	0.3528
o1-mini	10.20	1.12	0.30	3.142	0.3528
GPT-4o (Mar '25)	10.20	1.12	0.30	3.142	0.3528
GPT-4o mini	6.50	1.12	0.30	3.142	0.3528
GPT-4 Turbo	6.50	1.12	0.30	3.142	0.3528
GPT-4	6.50	1.12	0.30	3.142	0.3528
Claude-3.7 Sonnet	10.20	1.14	0.18	3.142	0.385
Claude-3.7 Sonnet ET†	10.20	1.14	0.18	3.142	0.385
Claude-3.5 Sonnet	10.20	1.14	0.18	3.142	0.385
Claude-3.5 Haiku	10.20	1.14	0.18	3.142	0.385
LLaMA-3.3 70B	10.20	1.14	0.18	3.142	0.385
LLaMA-3.2-vision 90B	10.20	1.14	0.18	3.142	0.385
LLaMA-3.2-vision 11B	10.20	1.14	0.18	3.142	0.385
LLaMA-3.2 3B	10.20	1.14	0.18	3.142	0.385
LLaMA-3.2 1B	10.20	1.14	0.18	3.142	0.385
LLaMA-3.1-40B	10.20	1.14	0.18	3.142	0.385
LLaMA-3.1-70B	10.20	1.14	0.18	3.142	0.385
LLaMA-3.1-8B	10.20	1.14	0.18	3.142	0.385
LLaMA-3-70B	10.20	1.14	0.18	3.142	0.385
LLaMA-3-8B	10.20	1.14	0.18	3.142	0.385

In short, artificial intelligence research nowadays is not merely interested in technical progress, but it is expected that such systems are ethically responsible, environmentally friendly and socially compliant.

As seen in Table 4, the critical power and carbon intensity values of the models are largely similar, but there are small differences in values such as PUE and WUE. These differences can be important in terms of environmental impact when making model selection.

4.    Results

In this study, we compared transformer-based models, starting with the initial models such as BERT, GPT, and DALL-E and extended the comparison to current systems such as GPT-4, GPT-4o, Claude 3, Gemini, LLaMA 3, Mistral, Whisper and Sora. The technical architectures of the models and their areas of use were compared in our study. The findings of this study were summarized in Table 5.

Table 5. Comparative Analysis of Transformer-Based Generative AI Models.

Model		Learning Structure	Contextual Ability	Mode of Output Generation	Ethical and Societal Sensitivity		Model Size (Parameter Count)
BERT	MLM (Masked Language Modeling), NSP		Bidirectional context	Encoded representation, classification output	Data bias, limitations in explainability	~110M (BERT-Base), ~340M (BERT-Large)
GPT-3/4	Autoregressive, causal attention		Unidirectional, long-context processing	Free-form text generation	Hallucination, issues with factual accuracy and citation	GPT-3: 175B; GPT-4: 1.75T (estimated)*
GPT-4o	Multimodal autoregressive learning		128K tokens, real-time analysis	Text + visual + audio output	Privacy concerns, energy efficiency, risks in voice-based interaction	Not officially disclosed
Claude 3	RLHF, Constitutional AI		200K tokens, coherent responses	Long-form text analysis	Ethical filtering, knowledge restriction	~ 175 B (estimated)*
Gemini	Multimodal (text, image, audio, code)		Conceptual + multimodal contextual understanding	Text, code, image, and audio generation	Safety filters, bias mitigation strategies	1 T+ parameters (Ultra*), 500 B+ parameters (Pro*)
Mixtral	Mixture of Experts (MoE)		Moderate contextual awareness	Fast generation of text and code	Open-source, low resource consumption	~12B active (MoE total ~65B)
DALL-E	CLIP-based text-to-image alignment		Visual-conceptual context	Image generation	Copyright issues, ethical constraints in content generation	DALL-E (original): 12B parameters
Whisper	Multilingual supervised learning on large-scale speech data		Multilingual, noise-resilient	Speech-to-text transcription, speech translation	Misrecognition of names, contextual errors, open-source adaptability	OpenAI offers different sizes: tiny (39M), base (74M), small (244M), medium (769M), large (1.55B)
Sora	Transformer + Diffusion-based hybrid learning		Narrative coherence and visual consistency	Text-to-video generation (visual + auditory simulation)	Deepfake risks, content authenticity, watermarking and safety concerns	Not officially disclosed; likely hundreds of billions to trillions

(*) These are estimations and does not reflect actual parameters, actual parameters might vary since they're not publicly disclosed by the organization

The findings of the review were a close look at the working mechanisms of the models, the nature of the outputs they produce, and the specific areas in which they are applied. The analysis recognized that, although each model possesses distinctive characteristics, there are also a number of characteristics they share in common. The findings are given below in four general dimensions:

4.1. Learning Structure

BERT provided a strong foundation for learning contextualized meaning through bidirectional masked language modeling, while GPT series provided natural and inventive text generation through autoregressive prediction. DALL-E performed image generation through text-image matching. Among the models, GPT-4 and GPT-4o were capable of processing text, visual and audio data and multimodal training procedures, while Claude 3 was meant to process long texts with great accuracy. Gemini, on the other hand, was multimodal trained to learn diverse forms of data such as text, images, audio, and video in the same setting. Open-source models such as LLaMA 3 and Mistral were modelled on novel training methods with high efficacy and flexibility due to their small size. Whisper was noisy audio-trained multilingual and possessed robust speech recognition, and Sora had a model that learned cross-modal relation between altering visual scenes and text descriptions and served as an alternative for text-to-video generation.

4.2. Contextual Ability

While BERT accurately captured the meaning relationships within the text through the use of bidirectional context, the GPT series stood out in producing more fluent, logically coherent texts. DALL-E could make sense of text annotations in visual space thanks to CLIP integration. Claude 3 stood out in multi-step reasoning by successfully maintaining context continuity in long text sequences, while Gemini had the ability to establish conceptual relationships between multiple data types simultaneously. LLaMA 3 was optimized for multilingual context understanding. Mistral models provided fast and accurate contextual analysis in systems with limited resources. While Whisper performed context adaptation across different languages and accents with high accuracy, Sora provided conceptual integrity by preserving semantic continuity across frames in video sequences generated based on textual inputs.

4.3. Mode of Production

There were also significant differences between the generation strategies of the models. While BERT was based on predicting masked words, GPT models performed left-to-right autoregressive generation. DALL-E performed multimodal generation that switched from text to visual. GPT-4o was capable of real-time generation by directly processing audio inputs, while Gemini could produce multimodal outputs. Claude 3 stood out especially with its consistency and accuracy-oriented production, while Mistral models stood out with their faster production capacity. Whisper was a multilingual, near-real-time transcription model that can converted spoken input into written text with high accuracy, while Sora offered next-generation multimodal production capabilities that combined visual coherence and temporal continuity by generating realistic video sequences from textual commands.

4.4. Ethical and Social Impacts

All models ran the risk of reflecting biases in the datasets they were trained on. Especially in GPT-based models, false information generation, so-called hallucination generation, was observed. In DALL-E, copyright and visual manipulation issues arose depending on the training images used. Privacy-related issues that could potentially arise with voice inputs in models such as GPT-4o and information security and auditing needs in long-context systems such as Claude 3 were identified. Further, the energy-intensive nature of model training introduced new environmental sustainability concerns.

These findings revealed how generative artificial intelligence systems based on transformer architecture were positioned in different contexts and what technical and ethical advantages and risks they brought with them.

With Whisper, transcribing audio data without permission in particular posed serious ethical risks in terms of data privacy and user confidentiality. Sora, on the other hand, brought legal and ethical discussions to the agenda in areas such as deepfake production, personality rights and copyright problems, with its video production capacity that could mimic real person likenesses or brand images.

5.    Conclusion

This paper provided a qualitative account of the important role of the transformer architecture within today's AI models and its implementation in various models. The discussion identified that each model had its own unique strengths, but also pointed to the ethical, social and environmental challenges they shared.

BERT caused a paradigm shift in the area of contextual semantics, and GPT has seen widespread application for its capacity to generate creative text. DALL-E caused a paradigm shift in the area of AI artwork generated from text-based to visual storytelling. One of the most impressive features of Claude 3 was its high degree of coherence and semantic consistency across a vast corpus of text data. Gemini had future-proofed design through its multimodal architecture and security-first design. LLaMA 3 and Mistral offered open-source, client-specific solutions that promoted increased accessibility and innovation. The relatively newer additions of Whisper and Sora also showed the increasing inroads of transformer-based models into real-time speech-to-text transcription and text-to-video synthesis, respectively.

However, concomitant with these technological advancements, concerns surrounding the utilization of big data, energy consumption, hallucination generation, algorithmic bias, and content ownership gained significant prominence. The pressing necessity to engineer AI systems that are ethical, transparent, and environmentally sustainable necessitated consideration from both technical experts and policymakers.

To assist practitioners and researchers in selecting and applying transformer-based models responsibly, a practical guide is proposed that summarizes typical areas of use, recommended models, their rationale, and key considerations. Table 6 is designed to help in relating model capabilities to requirements of applications, along with communicating knowledge of the associated risks and operational challenges.

BERT and RoBERTa are recommended for text understanding and search systems. Because these models capture the meaning in texts well and are successful in resolving ambiguities. However, big data and powerful computers are required to get good results.

GPT-4 and Claude 3 stand out for creative text generation and chatbots. These models produce natural and fluent text. They are well suited to interactive systems and narrative content creation, although practitioners must be aware of the risk of hallucinations wherein the models are able to generate plausible-sounding but inaccurate or misleading information and thus post-generation checking and, in some cases, human review are necessary.

Table 6. The table aims to guide practitioners in aligning model capabilities with specific use cases while being mindful of potential risks and operational challenges

Application Area	Recommended Model(s)	Rationale	Key Considerations
Text understanding / search systems	BERT, RoBERTa	High contextual semantic accuracy	Requires large-scale pretraining data and compute
Creative text generation / chatbots	GPT-4, Claude 3	Fluent and creative text production	Risk of hallucination; needs content validation
Visual content generation	DALL-E, Gemini	Strong for text-to-image / multimodal storytelling	Copyright, content ownership challenges
Speech-to-text / speech processing	Whisper	Real-time multilingual transcription	Accent and language coverage limitations
Text-to-video / multimodal synthesis	Sora, Gemini	Enables text-to-video and multimodal output	High compute and energy consumption
Open-source / customization	LLaMA 3, Mistral	Supports on-premise customization and transparency	Maintenance burden; requires in-house expertise

In image content creation, both DALL-E and Gemini are capable of translating text prompts into interesting images and multimodal stories. These capabilities help in aiding creative industries, advertising, and education. Their use, however, raises copyright, intellectual property rights, and ownership of content issues, which require careful management and policy guidance.

For speech-to-text, Whisper is recommended. It supports real-time and multilingual translation. Its accuracy in some languages and accents may be inconsistent, however. Sora and Gemini offer state-of-the-art text-to-video and multimedia generation solutions. These models are very demanding in terms of processing power and energy.

LLaMA 3 and Mistral enable flexible frameworks that foster openness and provide businesses with the capability to tailor models to their specific requirements.

Overall, here, not only is it necessary to boost the strength and usability of these models, but also to render their deployment secure, fair and compatible with long-term social and environmental well-being. The study underlines the importance of continued interaction and dialogue between developers, regulators and different segments of society.

6.      Declarations

Acknowledgement

Not applicable

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Authors` Contributions

All authors have participated in drafting the manuscript. All authors contributed equally to the manuscript and read and approved the final version of the manuscript.

Conflict of Interest

The authors certify that there is no conflict of interest with any financial organization regarding the material discussed in the manuscript.

Data Availability

This review is entirely dependent on previously published literature and does not incorporate any novel datasets. All data that support the findings of this study are available within the cited articles and publicly accessible databases.

Ethics Approval

No need for ethical committee approval.

Declaration of AI Usage

AI-assisted tools were employed in this study for minor tasks such as grammar correction, language refinement, and proofreading. These tools were used transparently and in a manner that does not compromise the authors’ intellectual contribution. The authors affirm that all substantive content reflects original thought and upholds academic integrity.

References

[1] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008. https://doi.org/10.48550/arXiv.1706.03762

[2] Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179–211. https://doi.org/10.1207/s15516709cog1402_1

[3] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

[4] Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2022). Efficient transformers: A survey. ACM Computing Surveys, 55(6), 1–28. https://doi.org/10.1145/3530811

[5] Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8, 842–866. https://doi.org/10.1162/tacl_a_00349

[6] Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., ... & Liang, P. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. https://arxiv.org/abs/2108.07258

[7] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140), 1–67. https://jmlr.org/papers/v21/20-074.html

[8] Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460. https://doi.org/10.48550/arXiv.2006.11477

[9] Qian, Y., Chen, S., Li, Z., Wu, J., Zhang, Z., Yu, M., & Xie, L. (2021). SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.07205. https://arxiv.org/abs/2110.07205

[10] Anthropic. (2025). Introducing Claude 4. https://www.anthropic.com/news/claude-4

[11] Google DeepMind. (2025). Gemini 1.5 technical report. https://deepmind.google/technologies/gemini

[12] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Scao, T. L. (2023). LLaMA 3: Open foundation models. arXiv preprint arXiv:2302.13971. https://arxiv.org/abs/2302.13971

[13] Mistral AI. (2023). Mistral and Mixtral models. https://mistral.ai/news/

[14] Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. https://doi.org/10.1109/72.279181

[15] Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010). Recurrent neural network based language model. In Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH 2010), 1045–1048.

[16] Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. https://arxiv.org/abs/1406.1078

[17] Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. https://arxiv.org/abs/1409.0473

[18] Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, 27, 3104–3112

[19] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

[20] Graves, A., Jaitly, N., & Mohamed, A. (2013). Hybrid speech recognition with deep bidirectional LSTM. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding (pp. 273–278). IEEE. https://doi.org/10.1109/ASRU.2013.6707742

[21] Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning (pp. 1310–1318). PMLR. http://proceedings.mlr.press/v28/pascanu13.html

[22] Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018). Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (pp. 284–294). https://doi.org/10.18653/v1/P18-1027

[23] Lake, B. M., & Baroni, M. (2018). Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning (pp. 2879–2888). PMLR. http://proceedings.mlr.press/v80/lake18a.html

[24] Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019). Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 2978–2988). https://doi.org/10.18653/v1/P19-1285

[25] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. https://arxiv.org/abs/2010.11929

[26] Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2023). Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, 5.

[27] Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450. https://arxiv.org/abs/1607.06450

[28] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770–778). https://doi.org/10.1109/CVPR.2016.90

[29] Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. https://arxiv.org/abs/1810.04805

[30] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. https://arxiv.org/abs/1907.11692

[31] Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. https://arxiv.org/abs/1910.01108

[32] Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). ALBERT: A lite BERT for self-supervised learning of language representations. International Conference on Learning Representations. https://openreview.net/forum?id=H1eA7AEtvS

[33] Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 4171–4186. https://doi.org/10.48550/arXiv.1810.04805

[34] Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8, 842–866. https://doi.org/10.1162/tacl_a_00349

[35] Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019). How to fine-tune BERT for text classification?. In China National Conference on Chinese Computational Linguistics (pp. 194–206). Springer. https://doi.org/10.1007/978-3-030-32381-3_16

[36] Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

[37] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://doi.org/10.48550/arXiv.2005.14165

[38] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

[39] Tamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021). Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503. https://doi.org/10.48550/arXiv.2102.02503

[40] Floridi, L., & Chiriatti, M. (2020). GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30, 681–694. https://doi.org/10.1007/s11023-020-09548-1

[41] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., ... & Sutskever, I. (2021). Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092. https://arxiv.org/abs/2102.12092

[42] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022). Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125. https://arxiv.org/abs/2204.06125

[43] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (pp. 8748–8763). PMLR.

[44] McCormack, J., Gifford, T., & Hutchings, P. (2019). Autonomy, authenticity, authorship and intention in computer generated art. Digital Creativity, 30(1), 52–68. https://arxiv.org/abs/1903.02166. https://doi.org/10.48550/arXiv.1903.02166

[45] Raji, I. D., & Buolamwini, J. (2019). Actionable auditing: Investigating the impact of public AI audits. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 429–435). https://doi.org/10.1145/3306618.3314244

[46] OpenAI. (2023). GPT-4 technical report. https://cdn.openai.com/papers/gpt-4.pdf

[47] OpenAI. (2024). Introducing GPT-4o: The Multimodal Future of AI. https://openai.com/index/gpt-4o-system-card/

[48] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. https://doi.org/10.48550/arXiv.2307.03172

[49] Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., ... Kaplan, J. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv preprint arXiv:2212.08073. https://doi.org/10.48550/arXiv.2212.08073

[50] Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., ... Gabriel, I. (2021). Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. https://doi.org/10.48550/arXiv.2112.04359

[51] Meta AI. (2024). Introducing LLaMA 3: The most capable openly available LLM to date. https://ai.meta.com/blog/meta-llama-3/

[52] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., ... & Scao, T. L. (2023). LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. https://arxiv.org/abs/2302.13971

[53] Mistral AI. (2023). Mistral 7B technical report. https://mistral.ai/news/announcing-mistral-7b/

[54] Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. https://doi.org/10.1145/3571730

[55] Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 1906–1919). https://doi.org/10.18653/v1/2020.acl-main.173

[56] Bolukbasi, T., Chang, K. W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. Advances in Neural Information Processing Systems, 29, 4349–4357.https://doi.org/10.48550/arXiv.1607.06520

[57] Bender, E. M., & Friedman, B. (2018). Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6, 587–604. https://doi.org/10.1162/tacl_a_00041

[58] Crawford, K., & Paglen, T. (2021). Excavating AI: The politics of training sets for machine learning. Excavating.ai. https://www.excavating.ai

[59] Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. https://arxiv.org/abs/1702.08608

[60] Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3645–3650). https://doi.org/10.18653/v1/P19-1355

[61] OpenAI. (2022). Introducing Whisper. https://openai.com/research/whisper

[62] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., & Krueger, G. (2022). Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356. https://arxiv.org/abs/2212.04356

[63] OpenAI. (2024). Sora: Creating video from text. https://openai.com/sora

[64] Jegham, N., Abdelatti, M., Elmoubarki, L., & Hendawi, A. (2025). How hungry is AI? Benchmarking energy, water, and carbon footprint of LLM inference. arXiv preprint arXiv:2505.09598. https://arxiv.org/abs/2505.09598

Author information

Copyright information