A Review Study on Energy Consumption in Cloud Computing

: Cloud computing has become a fundamental technology for a wide range of computing services, yet its increasing energy demands present substantial environmental and economic challenges. With the rapid growth of Cloud services and applications, increasing number of researches have been focused on energy saving. The need to reduce energy costs is a constant challenge of cloud providers and data centers. This paper offers an extensive review of the issues surrounding energy consumption in cloud computing, with a focus on algorithms associated with the situational awareness, consolidation, allocation, placement/migration, and scheduling of virtual machines and containers. We conduct a critical analysis of studies from 2018 to 2023, comparing various methodologies aimed at achieving energy efficiency without sacrificing performance. This review delineates current trends, identifies gaps in existing research, and proposes directions for future investigations. Our study emphasizes the necessity of cultivating sustainable practices in cloud computing and provides valuable insights into the practical implementation of energy-efficient solutions in cloud environments.


INTRODUCTION
Data centers are at the heart of modern software technology.They play a critical role in expanding the business world's ability to do much more with much less, both in terms of physical space, money and the time required to create and maintain data.As shown in Figure 1, the energy consumption source of the data center can be roughly divided into two parts [23]: energy consumption of IT facilities and infrastructure energy consumption.IT facilities refer to information technology facilities such as server systems (including servers and storage) and network systems, while the supporting infrastructure mainly includes cooling systems and power supply systems (including power supply systems and lighting systems).

Figure 1 Data center energy consumption source decomposition
Early versions of data centers are like isolated resource areas with physical servers, cooling systems and networking equipment used for enterprise backup systems or data storage such as silos.Technological developments are giving rise to new-generation data centers.One of the innovations is cloud computing, in which computing services such as programs, storage space, expert services, video games, films and music are made available on demand via an online service.Mobile development has also triggered a high demand for online services and energy [23].Cloud computing is founded on the principles of Service-Oriented Architecture (SOA) inherent in Web 2.0, along with the virtualization of both hardware and software resources.Virtualization technology provides more opportunities to data centers about giving adequate resources for computing and consuming less energy.Virtualization also offers data centers the opportunity to process more with the same infrastructure.Thus, virtualization and research on virtualization became popular.There are several other ways to reduce energy consumption related to information processing resources such as storage systems, cache management systems, processor architecture, schedulers, etc.But the popularity of virtualization motivates us to do this overview study.Rapid development of cloud computing technologies and applications caused cloud data centers have grown exponentially in recent years.One of the major problems with current cloud data centers is their huge energy consumption that makes management of energy consumption one of the hottest research topics.

Figure 2 Energy consumption forecast [2]
Figure 2 presents an "expected case" projection by Andres Andrae, who is a specialist in sustainable Information and Communication Technology (ICT).In his best scenario, datacenter energy consumption is growing 15 times higher during 2010 to 2030.Also energy consumption in wired networking is growing nearly the same.Data centers currently consume an estimated 200 terawatt hours (TWh) annually.This amount exceeds the national energy consumption of several countries, including Iran.It accounts for half of the global electricity used for transportation [1]. Figure 3 shows the energy supply including only wholesale colocation facilities in North America.In Europe, Latin America, and Asia-Pacific, total inventory includes both wholesale and retail colocation facilities.Inventory continues to climb, with Northern Virginia leading CBRE's global rankings.It has 2,132 MW (2.1 GW) of supply, increasing 19.5% year-over-year from Q1 2022 to Q1 2023.Data center supply grew year-over-year in Frankfurt, London, Amsterdam and Paris (FLAP) as providers work to meet the strong demand across most top European markets.There is 672 MW of inventory in Latin America as of Q1 2023, primarily across Brazil, Mexico, Chile and Colombia.The region has experienced significant growth over the past three years, with supply doubling since Q1 2020.Brazil has grown the fastest, with inventory up 127% from 2020 to 2022.It is also the largest, with around 67% of the region's inventory.Tokyo, Sydney and Singapore each now contain over a half-GW of live power capacity.Sydney's inventory jumped 30% year-over-year.Lack of power availability is a key emerging challenge facing operators in some Asia-Pacific markets.According to the report for European Commission [5], data centers in London consumes 386 MWs, in Frankfurt 199 MWs, in Amsterdam 166 MWs and in Paris 129 MWs. Figure 4 shows energy consumption detail in data centers in Germany.In general, all data centers have similar energy consumption values.When analyzing the reports defined above, we examined the power consumption of the servers, as this accounts for the largest share in the graph in Figure 4. Research into servers or computing power is a very good way of reducing the energy consumption of data centers.
In recent years, technological developments in servers have led to lower energy requirements than expected.Use of energy-efficient processors, memory and storage devices in cloud infrastructure provides companies to reduce overall energy consumption.In addition to such improvements that can be made in terms of hardware on the servers, software developments are also being made.In our study, we do not focus on the hardware equipment in the data center.Both VMs and containers benefit from energy-efficient hardware.Especially developments in server and processing technology in a way of virtualization, make cloud computing more attractive for researchers.
Virtual machines (VMs) and containers are increasingly being used in data center environments to improve resource utilization, reduce hardware costs, and simplify application deployment.However, the use of VMs and containers also leads to increased energy consumption, which can have significant financial and environmental impacts.Energy consumption in managing VMs and containers can be attributed to several factors, including inactive VMs and containers, over-provisioned resources, inefficient workload placement, and cooling and power distribution systems.Various algorithms and techniques have been proposed to optimize resource allocation, scheduling and power management in VM and container management.If the energy efficiency of VMs and containers is taken into account in management, data centers can achieve significant energy savings and reduce their carbon footprints.Similar to VM management, container management can also benefit from energy-efficient resource allocation, migration and scheduling.Containerization has emerged as an alternative to virtualization, offering similar benefits but with a lower overhead in terms of resource usage and energy consumption.Container management is a complex task involving multiple factors such as resource allocation, workload scheduling, placement or migration and situational power monitoring.In recent years, containerization has become increasingly popular in academic studies.
In light of the information provided in Figure 4, our study has been conducted with a focus on server energy consumption in cloud computing environments.This study specifically concentrates on evaluating algorithms related to virtualization and containerization within cloud computing.The focus is on how these algorithms impact CPU usage, which is the primary consumer of power in a server.These algorithms are thoroughly assessed from various perspectives, encompassing aspects such as situational awareness, scheduling, consolidation, allocation, and placement/migration.Cloud computing and virtualization methods are explained in Section II and the virtualization management system is provided in Section III.The used algorithms for situational awareness, scheduling, consolidation, allocation, and placement/migration of virtual machines and/or containers are examined in Section IV.

BREAKTHROUGH IN CLOUD COMPUTING
Existing cloud data centers are fully virtualized for service consolidation and energy reduction.Additionally, motivations for adopting virtualization encompass enhanced flexibility, dynamic resource allocation, and improved resource utilization.As a result, this paradigm has gained considerable attractiveness, leading to the development of various solutions over the years.Broadly, these solutions can be categorized into two types: hypervisor-based virtualization and Operating System (OS)-based virtualization methods.
Hypervisor-based methods are generally used and become traditional technique for many cloud providers.For example, Amazon Web Services (AWS) and Rackspace use the XEN Hypervisor [7] which has gained popularity because of its early open source inclusion in the Linux kernel, and is one of the most mature virtualization solutions available [8].The Kernel-based Virtual Machine (KVM) [9], a relatively new open source hypervisor-based system, has gained momentum and popularity in recent years [10].KVM has become a natural choice for Linux VMs because it is included in the upstream Linux kernel.Also there are VMWare [16] , VirtualBox [20] and XEN [21] in hypervisor-based virtualization.
Hypervisor-free OS-based virtualization such as container-based virtualization provides multiple isolated user spaces with only one running kernel instance, while hypervisor-based systems operate at the hardware abstraction level.Because this virtualization offers higher performance without the need to invoke virtual machines and guest operating systems, hypervisor-free OS-based virtualization systems are widely used by successful cloud providers such as Google.In recent years, most of cloud providers provide hypervisorfree OS-based virtualization choice to consumers.This choice offers new possibilities for easy provisioning and fast deployment environment.Hypervisor-free OS-based virtualization serves as a lightweight alternative to traditional hypervisor-based virtualizations.For example, Google has stated that each week they start over billion containers across all of its services [11].Several OS-based systems have been released, including Linux Container (LXC) [12], Docker [13], BSD Jails [14], and Windows Containers [15].Docker is the most widely used.
In virtualization there are two ways for presentation of a software interface to virtual machines, which are full-virtualization and paravirtualization.Paravirtualization presents a modified interface to virtual machines and achieves better performance [22] because applications are not running on an extra operating system.Full-virtualization presents higher compatibility by completely emulating the underlying hardware and this makes virtual machine like emulating a computer.This feature makes full-virtualization have more overhead than paravirtualization in because of running an extra full operating system on host operating system.

VIRTUALIZATION AND CONTAINERIZATION
Figure 4 clearly illustrates that within the realm of cloud computing, servers are responsible for the largest share of energy consumption.This underlines the significant impact that reducing server energy consumption can have on enhancing overall energy efficiency in cloud environments.It is important to acknowledge the extensive research and development efforts that have been dedicated to managing energy consumption, particularly focusing on Central Processing Unit (CPU) management and network management.
Virtualization allows multiple virtual machines (VMs) to share the same physical resources, which leads to better utilization of hardware and reduces the overall energy consumption.The rapid expansion in both number and scale of cloud data centers globally is a direct consequence of the increasing number of cloud computing users.This expansion has led to a more pronounced problem of energy consumption.The issue is multifaceted: it escalates operational costs and diminishes the return on investment for cloud service providers, while also contributing significantly to carbon dioxide emissions, thereby exacerbating global warming.Consequently, this problem has garnered considerable attention from both cloud service providers and governmental bodies.Containerization represents a methodology in software development and deployment that enables applications to operate reliably and consistently across various computing environments.This approach involves encapsulating an application along with all its dependencies into a unified package, known as a container.This container can then be executed on any system equipped with the requisite containerization technology.
A container provides a complete runtime environment for an application, including the operating system, libraries, and any other dependencies that the application needs to run.This allows developers to create applications that can be easily moved between different environments, such as development, testing, and production, without needing to modify the application code or its dependencies.Containerization technology is typically based on the use of container engines such as Docker, LXC, etc. , which allow containers to be built, shipped, and run on a wide range of platforms, including on-premises data centers, public cloud services, and edge computing devices.

Figure 5 CPU processing performance (Host and container) [18]
Virtual Machines (VMs) and containers are two widely utilized virtualization technologies, implemented at the hardware and Operating System (OS) levels, respectively.Consequently, containers can be deployed on VMs, allowing them to complement each other and function more effectively in tandem.
In Figure 5 you can see the performance of containers and hybrid use.Containers are lighter than VMs because they share the kernel of the host operating system, resulting in lower overhead and potentially lower energy consumption.They are also more portable and can be started and stopped more quickly, making them suitable for dynamic workloads.

COMPARISON OF VIRTUALIZATION AND CONTAINERIZATION
Virtualization and containerization are two different approaches to manage resources in data centers.Virtualization uses hypervisors to create multiple virtual machines, while containerization uses a single operating system kernel to create multiple containers.Both approaches offer benefits in terms of resource utilization and energy consumption.However, containerization has several advantages over virtualization in terms of energy consumption.Containerization has a lower overhead in terms of resource usage and energy consumption than virtualization.Containers use the same operating system kernel, which reduces memory footprint and CPU utilization compared to virtual machines.This results in a lower energy consumption with containerization compared to virtualization.Containerization also allows more efficient workload scheduling and resource allocation compared to virtualization.Containers can be quickly started and stopped, which allows for more flexible workload scheduling.Resource allocation in containers is also more efficient as they share the same operating system kernel, which allows more efficient resource utilization Virtual machine (VM) management refers to the process of creating, deploying, and managing virtual machines in a data center environment.VMs are software-based representations of physical hardware, which can be used to run multiple operating systems and applications on a single physical server.VM management involves the allocation of resources, such as CPU, memory, and storage, to VMs and the monitoring and optimization of VM performance [24].Container management bears similarities to VM management, but it employs a more lightweight and efficient approach for packaging and deploying applications, as opposed to using a full virtual machine.The process of container management encompasses the creation, deployment, and administration of containers, along with resource allocation and performance monitoring.By encapsulating an application and all its components within a container, it abstracts the underlying infrastructure and operating systems on which the application is deployed [25].

VIRTUALIZATION MANAGEMENT SYSTEM INTERNAL WORKFLOW
In recent years, several approaches have been proposed to optimize energy consumption in VM and container management.These approaches include dynamic workload consolidation, energy management and resource allocation algorithms.Dynamic workload consolidation involves moving VMs and containers between servers to consolidate workloads and power down unused servers.Power management is about reducing the power consumption of unused servers and components.Resource allocation algorithms aim to allocate resources efficiently by predicting the resource requirements of VMs and containers and allocating resources accordingly.
There are various algorithms that are used by virtualization managers in cloud environments.To categorize these algorithms, we need to determine the levels.
Cloud providers make agreements with their customers to define the service level, the so-called Service Level Agreements (SLA).These agreements define the requirements and priorities of the customer requests.These definitions are also used for processing customer requests for VM and container management.The virtualization managers prioritize the incoming tasks before processing the requests, and the requirements specified in the contract are also taken into account in this prioritization.

Fig.6 Workflow of a virtualization manager system decision stages for virtualization objects (VMS)
The virtualization management system comprises many subsystems and decides on the management of virtualization objects within the framework of 3 stages as seen in Figure 6.These stages are as follows: Pre-processing, decision making and support and action plans.
Under the heading of pre-processing, listener systems and queuing systems can be positioned to recognize future requests and requirements.Listener systems, as the name suggests, are always on standby and listen to identify potential requests.Requests and tasks that arrive after this stage are forwarded to the sorting systems to be sorted.Immediate metric measurements and prioritization information from the SLA, which are necessary for the ranking systems to do their job, are provided by the supporting systems.The integration of queuing and sorting processes can be efficiently achieved through the utilization of support systems.These systems are designed to handle complex tasks by organizing and prioritizing incoming data or requests, thereby enhancing overall operational efficiency.
In the decision-making and support process, performance metrics are generated based on the current situation from the virtualization system and the requirements stipulated in the service level agreements signed with customers.These metrics are then forwarded to the subsequent stage, which is the Sorting process, within the same operational workflow.During sorting, incoming requests are evaluated using SLA data and performance metrics, sorted using the specified algorithms and sent to the scheduler.
After the sorting process is completed, a decision action is processed by the management system.

Decisions can be; -Creating and/or placement of a virtualization component (VM or container) -Consolidation of virtualization component -Allocation of virtualization component -Migration of virtualization component
The decision action triggers the scheduling process to perform the selected action in the selected time.
These workflow elements contain a variety of orders for the execution of certain tasks.
The algorithms used in pre-processing process are used in decision-making and supporting and action plans.Artificial intelligence can be used, for example, in situational awareness or in the decision-making process.
Most studies focus on sorting, scheduling, consolidation, allocation and placement/migration processes.The remainder of this section is dedicated to providing an explanation of Virtual Machine System (VMS) processes.Subsequently, in Section IV, academic articles that have been thoroughly researched and presented are provided.a. Situational awareness: Cloud providers regularly monitor and analyze resource usage to identify inefficiencies and optimize resource allocation.Monitoring tools for VMs, such as VMware vSphere and Microsoft System Center, and for containers, such as Prometheus and Grafana, can provide insights into VM and container performance and resource usage.This information can be used to optimize VM allocation, ensuring that resources are used efficiently and energy consumption is minimized.b.Scheduling: Scheduling is a process for determining the right time to perform a certain action.We can either plan the timing ourselves or let AI determine it based on external parameters.In addition, the load distribution can be planned in this phase according to the incoming workload.c.Consolidation: Consolidating workloads onto fewer servers is an effective strategy for maximizing resource utilization and minimizing the number of idle servers, which continue to consume energy despite being underutilized.In the context of VMs, server consolidation can be achieved by deploying multiple Virtual Machines on a single physical host.This approach reduces the necessity for numerous physical servers to host applications, thereby leading to decreased energy consumption.Regarding containers, their inherently lightweight nature allows for even higher consolidation ratios.Operating multiple containers on the same host can result in enhanced resource utilization and further reductions in energy consumption.d.Allocation: Allocation in virtual machine (VM) management is a process that involves distributing virtual components across physical systems.A key method for reducing energy consumption in this context is through effective resource allocation and scheduling.This process entails dynamically assigning resources, like CPU and memory, to VMs, taking into account their specific workload and performance needs.To optimize resource allocation and scheduling in VM management, various algorithms have been developed and proposed.e. Placement/Migration: Placement is the task of first placement of a virtual component to the proper physical/virtual environment.Migration is the task of moving a virtual component to another physical or virtual environment.Especially migration process is a costly process to handle.

LITERATURE REVIEW
Optimizing energy consumption when managing VMs and containers requires the use of various algorithms that enable efficient resource allocation, workload management, and optimization of cooling and power distribution systems.In this section, we explore and discuss several main algorithms that recent research has proposed, focusing on optimizing energy consumption in the management of Virtual Machines (VMs) and containers.
In our research, we conducted an analysis of sources from Scopus, IEEE, MDPI, Wiley, Springer, Semantic Scholar, Google Academic, ACM, and various cloud computing-related conferences.The selection criteria for articles or papers were as follows: a. Relevance to cloud computing, explicitly excluding those containing keywords such as 'ship', 'heat', 'solar', and 'package'.b.Publication date between 2018 and 2023.c.Inclusion of keywords related to 'Virtual' or 'Container'.d.Mention of 'power usage' or 'energy consumption'.e. Exclusion of publications focusing on 'network', 'edge', 'IoT', '5G', and '6G'.The rationale behind these criteria included: a.To capture the trends in publications over the last five years that are pertinent to energy efficiency in cloud computing.b.To distinguish our focus from maritime and transportation publications.c.To concentrate on algorithms that are directly related to energy efficiency in virtual machines and containers, with a specific emphasis on CPU usage, which is our primary area of focus.d.To segregate our research from publications primarily concerned with network, edge computing, and hardware-related topics.Our mapping study resulted in a knowledge base of current research approaches, methods, techniques, best practices and experiences used in cloud computing, with a particular attention to virtualization management system processes as situational awareness, scheduling, consolidation, allocation and placement/migration.

SITUATIONAL AWARENESS
In this process, system is regularly monitored and analyzed in terms of resource usage to identify inefficiencies and optimize resource allocation.Monitoring and situational awareness (SA) are crucial aspects of ensuring the health, security, and performance of cloud computing environments.Recent years have seen significant advancements in algorithms designed to collect, analyze, and interpret data for effective cloud monitoring and awareness.By leveraging the advancements in machine learning and other technologies, these algorithms are • Email: editor@ijfmr.com

IJFMR240111927
Volume 6, Issue 1, January-February 2024 10 becoming increasingly sophisticated and effective in ensuring the health, security, and performance of cloud environments.Here's specific monitoring and awareness areas: • Security-aware Monitoring: This area focuses on monitoring for security threats and vulnerabilities in cloud environments.Papers like [47] discuss techniques for anomaly detection and intrusion prevention in cloud systems.• Performance Monitoring: This area focuses on monitoring cloud resources and applications to ensure optimal performance.Papers like [28] address resource utilization and performance bottlenecks in cloud environments.• Cost Optimization Monitoring: This area focuses on monitoring cloud resource usage and costs to optimize spending.Papers like [26] discuss techniques for predicting resource demands and optimizing resource allocation for cost efficiency.Table 1 is a curated review of relevant papers published in the past five years.The table shows focus, algorithms and key features of situational awareness/monitoring studies between 2018-2023.A study [29] proposes a cloud-based framework for space situational awareness (SSA), leveraging cloud computing's scalability and flexibility for tracking and monitoring space objects.It highlights the benefits of cloudbased SSA for efficient data storage, processing, and analysis.Another study [44] proposes a reinforcement learning-based approach for resource management in cloud data centers.Predicting the power consumption attributable to a specific virtual component presents a significant challenge.In study [35], two power models have been developed that correlate the workload of a Virtual Machine (VM) with its proportion of power consumption.These models are based on the power profiling of a server.The proposed models demonstrate an improvement in accuracy, approximately by 3%, compared to other existing models in the field.
To tackle the issue of inefficient resource utilization, existing approaches primarily concentrate on Virtual Machine (VM) allocation and migration.This strategy typically results in optimization at the Physical Machine (PM) level.However, other methods employ horizontal auto-scaling, which may not be an effective solution, particularly in the context of Infrastructure as a Service (IaaS) public cloud environments.The study proposes an approach that involves customizing the size of user Virtual Machines (VMs) to align with the resource requirements of their application workloads.This approach is based on an analysis of real backend traces collected from a VM operating in a production data center.The approach detailed in [6] involves allocating fixed-size resources to a Virtual Machine (VM) that are specifically designed to meet the application workload's demands.In cases where the demands exceed this predetermined resource allocation, the strategy utilizes vertical VM auto-scaling to manage the excess requirements effectively.This method aims to reduce energy consumption by Physical Machines (PMs) through enhanced resource utilization.Experimental results, derived from a simulation conducted on CloudSim Plus and utilizing GWA-T-13 Materna real backend traces, demonstrate that efficient resource utilization can lead to a reduction in data center energy consumption by approximately 40-52%.The research referenced in [17] introduces two Dynamic Voltage and Frequency Scaling (DVFS)-enabled host selection algorithms for Virtual Machine (VM) placement in a cluster, specifically the Carbon and Power-Efficient Optimal Frequency (C-PEF) algorithm and the Carbon-Aware First-Fit Optimal Frequency (C-FFF) algorithm.These algorithms are primarily designed to achieve two key objectives: to evenly distribute the load across servers and to dynamically adjust the cooling load in response to the current workload.
A notable aspect of these algorithms is their cluster selection strategy, which is based on both static and dynamic Power Usage Effectiveness (PUE) values, as well as the Carbon Footprint Rate (CFR).Furthermore, the research extends the cluster selection approach to encompass non-DVFS host selection policies.This extension includes the development of the Carbon-and Power-Efficient (C-PE) algorithm, Carbon-Aware First-Fit (C-FF) algorithm, and Carbon-Aware First-Fit Least-Empty (C-FFLE) algorithm.The findings from this research reveal that the C-FFF algorithm is particularly effective, achieving a 2% greater power reduction compared to the C-PEF and C-PE algorithms.This positions C-FFF as a powerefficient solution for reducing carbon dioxide emissions, while maintaining the same Quality of Service (QoS) as its counterparts.Additionally, it does so with lower computational overheads, further emphasizing its efficiency.

SCHEDULING
The dynamic and distributed nature of cloud computing necessitates efficient scheduling algorithms to optimize resource utilization, job completion times, and overall system performance.Table 2 shows focus, algorithms and key features of scheduling studies between 2018-2023.The study [64] proposes an ant colony optimization-based algorithm that schedules tasks across heterogeneous cloud resources while minimizing energy consumption.Another study [34] leverages reinforcement learning to schedule workflow tasks in the cloud, considering deadline and budget constraints.The study [61] addresses fairness concerns in multi-tenant cloud environments by proposing a priority queue-based scheduling approach that guarantees a minimum QoS for all tenants.The paper [67] explores dynamic voltage scaling for scheduling tasks with approximation tolerance in cloud datacenters, achieving energy savings without compromising accuracy.The study [68] investigates cost-efficient scheduling of containerized workloads in the cloud by considering resource costs and execution time.Another study [63] proposes a priority-aware scheduling algorithm for serverless functions in edge computing environments, ensuring timely execution of critical tasks.The paper [65] combines reinforcement learning and heuristic techniques for scheduling tasks and allocating resources in cloud environments, leading to enhanced performance.The study [66] proposes a federated learning framework for scheduling tasks in dynamic cloud environments, enabling distributed and collaborative learning among scheduling agents.

. Scheduling studies between 2018-2023 in respect of energy consumption
In a study [60], researchers propose an ant colony optimization-based algorithm that schedules tasks across heterogeneous cloud resources while minimizing energy consumption.The study introduces a novel approach for migrating Virtual Machines (VMs) utilizing a capuchin search algorithm (CapSA).This proposed method aims to leverage the strengths of both migration and scheduling.It does so by employing a hybrid approach that combines multi-objective CapSA and inverted ant colony optimization (IACO) algorithms.The approach involves selecting an optimal algorithm for subsequent tasks by using a decision-making framework that considers the specific conditions of the received tasks.Compared to previous methods, the proposed approach demonstrates superior performance in terms of energy consumption (EC), execution time (ET), and load balancing, showing improvements in the range of 15-20%.
In study [61], researchers introduced a time and energy-aware two-phase scheduling algorithm, named Best Heuristic Scheduling (BHS), specifically designed for scheduling Directed Acyclic Graphs (DAG) on processors in cloud data centers.This algorithm operates in two distinct phases: firstly, it sorts, and then it identifies the best performing action.The findings from this study indicate that the BHS algorithm achieves 19.71% more energy savings compared to the Multiheuristic Resource Allocation (MHRA) algorithm.Additionally, the makespan, which is the total duration required to complete execution, is reduced by 56.12% in heterogeneous environments, demonstrating the algorithm's effectiveness in optimizing both energy efficiency and execution time.
In study [34], researchers developed two workflow scheduling algorithms that leverage the structural properties of workflows.The first algorithm, named Structure-based Multi-objective Workflow Scheduling with an Optimal instance type (SMWSO), introduces a novel method for determining both the optimal instance type and the optimal number of Virtual Machines (VMs) to be provisioned.Additionally, in the Structure-based Multi-objective Workflow Scheduling with Heterogeneous instance types (SMWSH), the algorithm is adapted to include the use of heterogeneous VMs, thereby demonstrating its effectiveness in heterogeneous environments.Simulation results from this study indicate that the proposed algorithms achieve better energy efficiency in 80% of the tested workflow/workload scenarios.They also show a significant energy saving, exceeding 50%, when compared to a recent state-of-the-art algorithm, underscoring their potential for substantial energy conservation in cloud computing environments.
In study [41], a survey is conducted on resource scheduling methods, with a particular emphasis on Deep Reinforcement Learning (DRL)-based scheduling approaches in cloud computing.The study comprehensively reviews the application of DRL in this context and also delves into the challenges associated with it.Additionally, it discusses prospective future directions for the application of DRL in the scheduling of cloud computing resources.This examination offers a detailed insight into the evolving role of DRL in optimizing and streamlining scheduling processes within cloud environments.
In study [65], researchers propose a novel technique named RATS-HM (Resource Allocation and Task Scheduling using Hybrid Machine Learning) for combined resource allocation and efficient task scheduling in cloud computing, aimed at overcoming existing challenges.This technique includes several key components: First, it features an improved cat swarm optimization algorithm-based short scheduler for task scheduling, known as ICS-TS, which is designed to minimize make-span time while maximizing throughput.Secondly, it incorporates a group optimization-based deep neural network (GO-DNN) that facilitates efficient resource allocation, taking into account various design constraints such as bandwidth and resource load.Thirdly, the technique introduces a lightweight authentication scheme called NSUPREME for data encryption, enhancing the security of data storage.Finally, to establish the effectiveness of the RATS-HM technique, simulations are conducted in different setups, and the results are compared with state-of-the-art techniques.Notably, these simulations demonstrate that the proposed technique significantly reduces power consumption, achieving a reduction of nearly 68%, thereby highlighting its potential in enhancing energy efficiency in cloud computing environments.Workload scheduling is another critical factor in VM management, as it affects the performance and energy consumption of VMs.Several algorithms have been proposed to optimize workload scheduling in VMs.In [37], the authors proposed an algorithm that schedules VMs based on their resource usage and energy consumption.The algorithm uses a genetic algorithm to find the optimal schedule of VMs, which reduces energy consumption and improves performance.Several algorithms have been proposed to optimize workload scheduling in containers.In [56], the authors proposed an algorithm that schedules containers based on their resource usage and energy consumption.The algorithm uses a genetic algorithm to find the optimal schedule of containers, which reduces energy consumption and improves performance.

CONSOLIDATION
Consolidation involves moving VMs and containers across servers to consolidate workloads and turn off idle servers.This approach is based on the observation that many servers in data centers are often idle, leading to wasted energy consumption.By consolidating workloads and turning off idle servers, dynamic workload consolidation can reduce energy consumption in VM and container management.Consolidation algorithms are pivotal in optimizing resource utilization and reducing energy consumption within cloud computing environments.These algorithms effectively tackle the issue of underutilized resources.They achieve this by dynamically migrating and packing Virtual Machines (VMs) onto a smaller number of physical machines.This strategy leads to a decrease in the total count of active machines, subsequently lowering their collective energy footprint.This approach not only enhances efficiency but also contributes significantly to the sustainability of cloud computing operations.Table 3 shows focus, algorithms and key features of consolidation studies between 2018-2023.The study [70] proposes a DVS-based consolidation algorithm that balances energy savings with performance requirements.Another study [40], introduces a two-phase algorithm considering both energy consumption and server temperature to prevent overheating.In the study [55] the ant colony metaheuristic is utilized to schedule VMs while minimizing energy consumption and migration overhead.The study [52] ensures timely execution of critical tasks by incorporating deadline guarantees into the consolidation process.
Another study [51] addresses fairness concerns by allocating resources among tenants proportionally to their usage and priorities.The study [49] considers both execution time and resource costs when scheduling containerized workloads for optimal performance and budget efficiency.Another study [48] employs deep learning to predict resource demands and dynamically consolidate VMs for improved performance and resource utilization.The study [65] which is applied for situational awareness and monitoring is also used to combine reinforcement learning and heuristics to optimize scheduling and resource allocation in cloud environments.In another study, referenced as [66]  Optimizes execution time and resource costs for containerized workloads [48] Machine Learning Deep reinforcement learning Predicts resource demands and dynamically consolidates VMs for improved performance and utilization [65] Machine Learning (Hybrid) Hybrid reinforcement learning Combines reinforcement learning and heuristics for optimal scheduling and resource allocation [66] Machine Learning (Federated) Federated learning Enables collaborative learning among scheduling agents for dynamic cloud resource management Table 3

. Consolidation studies between 2018-2023 in respect of energy consumption
In study [42], a new approach is proposed to address the consolidation problem of Virtual Machines (VMs) with the aim of optimizing their energy consumption.This approach includes the introduction of a novel algorithm, complemented by an energy consumption model specifically designed to aid the algorithm in identifying optimal solutions.Experimental results from this study indicate that significant reductions in energy consumption can be achieved through dynamic VM consolidation.When compared with other classic algorithms, the proposed algorithm demonstrates superior performance, achieving an average energy consumption savings of 42%.This highlights its effectiveness in enhancing energy efficiency in VM management.The research presented in [32] introduces an Energy-Efficient Strategy (EES) designed for consolidating Virtual Machines (VMs) in a cloud environment.The primary objective of this strategy is to achieve a reduction in energy consumption while concurrently handling an increased volume of tasks with the highest possible throughput.Central to this proposal is the utilization of the performance-to-power ratio, which is employed to establish upper thresholds for overload detection.Additionally, EES takes into account the overall data center workload utilization to determine lower thresholds, an approach aimed at minimizing the frequency of virtual machine migrations.Simulation results from the study indicate that EES effectively leads to energy-efficient workload consolidation.This is achieved with a minimal number of migrations and a reduction in overall energy consumption.The results further suggest that EES is capable of saving energy without compromising the requirements of the user's workload.Specifically, the strategy is shown to reduce energy consumption by 19%, demonstrating its effectiveness in enhancing energy efficiency in cloud computing environments.In the approach proposed in [33], the methodology begins by partitioning Physical Machines (PMs) into distinct groups according to their load levels.Following this, a coalitional-game-based VM consolidation algorithm, referred to as CGMS, is employed.This algorithm is instrumental in selecting members from these groups to form efficient coalitions.Subsequently, VM migrations are executed among the members of these coalitions to maximize each coalition's payoff.Additionally, the approach involves shutting down PMs that exhibit low energy efficiency, further optimizing the overall system performance.Experimental results based on multiple cases clearly demonstrate that the proposed approach achieves higher energysaving (32.30% higher than Sercon in three scenarios on average).Hypervisors like VMware ESXi and Microsoft Hyper-V have built-in power management features that can reduce energy consumption.These features include dynamic voltage and frequency scaling, which adjusts processor power consumption based on workload demands.A method about energy-efficient VM management is power management, which involves dynamically adjusting the power consumption of servers and VMs based on workload demand.Several algorithms have been proposed to optimize power management in VMs.In the study referenced as [24], the authors explore the advantages of virtualization in the context of power management within data centers.They propose a power-aware Virtual Machine (VM) consolidation algorithm.This algorithm is designed to consolidate VMs onto a fewer number of physical servers, aiming to reduce energy consumption while simultaneously adhering to performance requirements.In [38], the authors proposed an algorithm that optimizes the power management of VMs using a reinforcement learning approach.The algorithm learns the optimal power management strategy for VMs, which reduces energy consumption and improves performance.

ALLOCATION
Resource allocation algorithms play a crucial role in cloud computing, determining how resources like CPU, memory, and storage are assigned to competing tasks and applications.As cloud environments evolve, researchers are constantly developing new and optimized allocation algorithms to address specific challenges and objectives.Researchers are concentrating their studies on specific areas of allocation, which include focusing on energy consumption, resource distribution, and budgetary limitations.Table 4 shows focus, algorithms and key features of allocation studies between 2018-2023.The study [58] proposes an ant colony optimization-based approach for heterogeneous cloud environments, prioritizing energy savings.Another study [61] introduces a priority queue-based algorithm that guarantees a minimum QoS for tenants in multi-tenant cloud environments.The study [60] integrates deadline constraints into the allocation process, prioritizing timely execution of critical tasks.The study [62] considers both execution time and resource costs when scheduling containerized workloads, aiming for cost-efficient performance.The study [34] addresses deadline and budget constraints simultaneously for workflow tasks using reinforcement learning.Another hybrid approach [65] combines reinforcement learning and heuristics for joint task scheduling and resource allocation.Another study [66]  In [54], the authors proposed an algorithm that optimizes the allocation of CPU and memory resources in containers.The algorithm uses a genetic algorithm to find the optimal allocation of resources, which reduces energy consumption and improves performance.In [34], the authors proposed an algorithm that dynamically adjusts the CPU frequency of containers based on their workload.The algorithm monitors the CPU utilization of containers and adjusts the CPU frequency accordingly, which reduces energy consumption without affecting the performance of containers.A recent solution in the field of cloud computing involves allocating high-priority Virtual Machine (VM) requests in close proximity, while other requests are assigned using the First Fit Decreasing (FFD) method.However, this approach does not distinguish between different types of requests by partitioning the hosts.Addressing this gap, study [19] introduces the Priority-Aware Resource Allocation (PARA) algorithm.PARA categorizes VM requests into three types: highly critical, critical, and normal.Correspondingly, it partitions the hosts into three levels, allowing high-priority VM requests access to a relatively larger pool of hosts compared to lower-priority requests.To evaluate the effectiveness of the PARA algorithm, simulations were conducted with 10,000 to 100,000 VM requests and 20 to 30 hosts.These were then compared with a non-partitioned pool of hosts using the FFD method, focusing on allocation rate and weighted score in relation to serving requests.The study specifically calculated the total allocated differences between PARA and FFD for both 20 and 30 hosts.The results show that PARA offers a more favorable total allocated difference, where the ratio is y:x:1, with y > 1 and x ≥ 1, indicating its superior performance in effectively allocating resources according to the priority of requests.
In study [30], researchers developed a genetic heuristic search optimization technique for dynamic consolidation of Virtual Machines (VMs), which is based on adaptive utilization thresholds.This approach aims to ensure a high level of compliance with Service Level Agreements (SLA).The strategy involves setting upper and lower utilization thresholds for hosts, with the primary goal being to maintain the total CPU utilization by all VMs within these dynamically fluctuating thresholds.To address the challenges of the dynamic virtual machine allocation policy, the power-aware schedulingbased resource allocation method, termed G-PARS, was proposed.This method was designed to optimize the allocation of VMs, focusing on energy efficiency and quality of service (QoS).Experimental results from the study indicate that the G-PARS strategy outperforms particle swarm optimization strategies in several aspects.It not only maintains high QoS but also results in lower energy consumption.Additionally, G-PARS demonstrates a significant advantage in reducing the number of active hosts, particularly under workloads that are life-threatening or critically demanding.In terms of power reduction, G-PARS shows an improved performance in reducing the energy consumption of each data center, with reductions ranging between 13-22% compared to existing algorithms.This highlights the effectiveness of G-PARS in achieving both energy efficiency and high-quality service in cloud computing environments.

PLACEMENT/MIGRATION
Placement and migration algorithms play a crucial role in optimizing resource utilization and performance in cloud computing.They determine where virtual machines (VMs) should be placed and when they should be migrated between physical machines to ensure efficient resource allocation, meet service level agreements (SLAs), and minimize operational costs.Table 5 shows focus, algorithms and key features of placement/migration studies between 2018-2023.A study [48] leverages deep reinforcement learning to predict resource demands and dynamically place VMs for improved performance and resource utilization.It demonstrates the effectiveness of DRL in proactive cloud resource management.Another study [55] proposes an ant colony optimization-based approach for heterogeneous cloud environments, prioritizing energy savings while considering resource heterogeneity.
A study [40] introduces a two-phase algorithm that prioritizes energy savings while preventing overheating in cloud environments, addressing thermal concerns during VM migration.An approach proposed in [59] is VMP-A3C, which is a dynamic placement algorithm based on A3C via migration.the purpose of placement is to find the best mapping between VMs and HMs, considering some constraints and objectives.In [36], the authors proposed an algorithm that optimizes the placement of VMs on physical servers to reduce energy consumption.The algorithm uses a heuristic approach to find the optimal placement of VMs, which reduces energy consumption without affecting the performance of VMs.
In study [39], researchers introduced a novel approach to VM consolidation, which takes into account both the current and anticipated future utilization of resources.This approach is operationalized through two key components: the host overload detection (UP-POD) and host underload detection (UP-PUD).A Gray-Markov-based model is employed to accurately predict future resource utilization.The effectiveness of this proposed approach was tested using real-world workload traces in CloudSim, and the results were benchmarked against existing algorithms.The simulations demonstrated that the approach significantly reduces both the number of VM migrations and energy consumption, while still maintaining the Quality of Service (QoS) guarantee.
In terms of energy consumption, the proposed approach achieved an average reduction of 42.7%, 38.1%, 39%, and 33.1% compared to the THR (Thresholding), IQR (Interquartile Range), MAD (Median Absolute Deviation), and LR (Local Regression) approaches, respectively.This substantial improvement is attributed to the reductions in both energy consumption and the Service Level Agreement Violation (SLAV) rate.The results are particularly noteworthy as they indicate that the approach effectively balances the trade-off between power cost and the assurance of QoS.This balance is crucial in cloud computing environments, where both energy efficiency and service reliability are key priorities.Also, power management techniques can be applied to container management.In the study [27], the authors proposed a power management framework for containerized applications that uses dynamic voltage and frequency scaling (DVFS) to adjust the power consumption of CPUs based on workload demand.The framework also includes a container migration algorithm to balance the energy consumption of different physical servers.In [57], the authors proposed an algorithm that optimizes the placement of containers on physical hosts to reduce energy consumption.The algorithm uses a heuristic approach to find the optimal placement of containers, which reduces energy consumption without affecting the performance of containers.

CONCLUSION
Figure 4 shows that the maximum energy consuming equipment in cloud computing are servers.Virtualization allows multiple virtual machines (VMs) to share the same physical resources, which leads to better utilization of hardware and reduces the overall energy consumption.Containerization also facilitates improvements in various aspects of technology management.It enhances application portability, accelerates the cycles of application development and deployment, and leads to better resource utilization and scalability.This is because containers can be managed and orchestrated more efficiently compared to traditional virtual machines.In Section IV, we discussed various algorithms proposed in recent research for optimizing energy consumption in VM and container management.These algorithms include dynamic workload consolidation, power management, resource allocation, and container orchestration.These algorithms can efficiently allocate resources, manage workloads, and optimize cooling and power distribution systems to reduce energy consumption in VM and container management.The selection of a suitable algorithm depends on the specific requirements of the system and the workload characteristics.Furthermore, the effectiveness of these algorithms can be further enhanced by combining them with other optimization techniques, such as workload prediction, resource provisioning, and power management.There is a novel approach within the domain of virtualization management that proposes the integration of multiple stages.For instance, some studies have introduced algorithms that simultaneously encompass situational awareness and allocation, representing a consolidated strategy under the umbrella of the virtualization manager.Container orchestration involves managing the deployment, scaling, and monitoring of containerized applications.This approach is based on the observation that containerization can reduce energy consumption by reducing the overhead of virtualization.By efficiently managing containerized applications, container orchestration algorithms can further reduce energy consumption in VM and container management.In light of these findings, our survey-based research further investigates methods for optimizing CPU utilization, with a specific focus on the roles of virtualization and containerization in cloud computing.The survey encompassed a diverse range of studies and expert opinions, revealing an emergent trend in the increased research focus on containerization technologies.This shift indicates a growing interest in exploring containerization as an alternative or complementary approach to traditional virtualization, especially in the context of its potential to improve energy efficiency in server operations.Containerization's increasing prominence and its potential to offer enhanced efficiency and scalability, which are crucial for reducing energy consumption in cloud computing servers.In summary, containerization is emerging as a key technology in cloud computing, offering enhanced efficiency and scalability.These features are instrumental in reducing energy consumption in cloud computing servers, thereby addressing one of the major challenges in the field.The shift towards containerization indicates a significant step towards more sustainable and energy-efficient cloud computing practices.This comprehensive survey thus provides valuable insights into current trends and future directions in CPU utilization strategies, particularly in the rapidly evolving field of cloud computing.

Table 1 . Situational awareness/monitoring studies between 2018-2023 in respect of energy consumption
It demonstrates the effectiveness of RL (Reinforcement Learning) in optimizing resource allocation for performance and energy efficiency.
, collaborative learning is enabled among scheduling agents within a dynamic cloud environment, facilitating efficient resource management.