ISG Provider Lens™ Analytics Services - Data Engineering Services – Specialist - U.S. 2023

Analytics investments in data and cloud are the cornerstones for deploying and scaling Generative AI.

The year 2022 left a marked impact on the global and U.S. analytics market with Generative AI (GenAI) and large language models (LLMs) getting unprecedented attention and playing a substantial role in driving growth. In 2023, enterprises and providers were making investments and taking initiatives to convert the GenAI hype into reality by generating the necessary momentum. This momentum, with the backdrop of enterprises increasingly relying on data-driven decision-making, further propelled the growth of the analytics market in the U.S. Ongoing economic uncertainty combined with reduced consumer activity has pushed enterprises to seek new ways of generating revenue, based on strong data-led business decisions.

According to ISG Research, 85 percent of enterprises believe that investment in GenAI in the next 24 months is important or critical. ISG’s State of Applied Generative AI Market Report details that GenAI will witness a four-phased maturation process — current adoption is still within the first two phases. In 2023, adoption has begun with knowledge management and functional process optimization, leading to product and offerings transformation (development of true AI-first products) and complete business transformation (reinvented
operational models entirely built around AI).

Such developments also indicate that analytics is becoming the foundation for the successful development and implementation of GenAI use cases. Some of the use cases in which data and analytics are gaining traction include:

• Data extraction
• Contextual searches and indexes
• Virtual assistant solutions
• Predictive analytics
• Performance analytics
• Recommendation engine
• Synthetic data generation

Moreover, there is a renewed focus on data from a GenAI perspective, where it is becoming increasingly critical for business transformation and reinvention. In the past, data availability and accessibility was viewed from an analytics perspective but with GenAI, the focus has shifted to data being viewed under the lens of
business value. Cybersecurity, privacy and the responsible use of data are becoming the key concerns for enterprises, driven by customer need to protect personal data.

Some of the key challenges enterprises face with analytics initiatives and projects include:

1. Recession impacting budget — no investments on new solutions or technologies

2. Deriving more value from existing analytics investments

3. Adopting the right strategies to become data-centric and evolving from being data-aware to becoming data-reliant

4. Ability to scale analytics and AI investments across an enterprise

5. Lack of a data-centric culture serves as a barrier in data democratization and monetization

The ISG Provider Lens™ Analytics Services study reveals that enterprises across the large and midmarket have different concerns and invest in varied analytics solutions in alignment with their digital and data maturity. It is imperative that enterprises engage with suitable service providers that exhibit a thorough understanding of enterprise-specific challenges and have the ability to prioritize and handle them effectively.

These challenges highlight the varied requirements of implementing AI and analytics solutions in diverse business environments, with due diligence ranging from technical aspects to organizational dynamics. Some of the key challenges identified for both large businesses and the midmarket with the analysis undertaken for the study include:

Data quality and integration

• Large Businesses: Struggling to handle extensive and diverse datasets while maintaining data quality and integrating information from various sources to ensure consistency and reliability.

• Midmarket: Limited data resources leading to data silos and the criticality of data accuracy for effective insights using analytics.

Costs and RoI

• Large Businesses: Upfront costs of acquiring and implementing advanced analytics solutions, integration expenses and the challenge of demonstrating a substantial RoI.

• Midmarket: Balancing limited budgets and cost-effective analytics solutions and demonstrating tangible value to justify the investment.

Talent and skills gap

• Large Businesses: Attracting and retaining skilled data scientists, data engineers and analysts, and building a robust analytics team with diverse expertise.

• Midmarket: Hiring specialized talent due to budget constraints and relying on existing staff to handle  analytics tasks.

Data security and privacy

• Large Businesses: Navigating complex data security and compliance requirements, protecting sensitive information from breaches and ensuring adherence to regulatory standards.

• Midmarket: Addressing cybersecurity concerns with limited resources and ensuring compliance with  relevant data privacy regulations.

Integration with existing systems

• Large Businesses: Ensuring seamless integration with complex existing IT infrastructures and legacy  systems and avoiding disruptions in ongoing operations.

• Midmarket: Overcoming potential compatibility issues with existing software and technology to ensure a smooth integration process.

Scalability

• Large Businesses: Ensuring that AI and analytics solutions can scale to accommodate growing data  volumes, expanding business needs and increasing user demands.

• Midmarket: Balancing the need for scalable solutions with current business size and future growth projections and avoiding overinvestment or underinvestment.

Change management

• Large Businesses: Managing resistance to change and ensuring effective adoption of analytics tools by employees at various levels.

• Midmarket: Adapting to new analytics processes and tools with a small workforce and ensuring a smooth transition through effective change management.

Understanding business needs

• Large Businesses: Aligning AI and analytics solutions with diverse business units and varied operational requirements, ensuring relevance and usability across an organization.

• Midmarket: Defining and understanding specific business needs to avoid overinvestment in or  underutilization of analytics tools and ensuring targeted benefits.

Real-time analytics

• Large Businesses: Implementing realtime AI and analytics solutions for timely decision-making in a complex organizational structure, which involves managing data streams efficiently.

• Midmarket: Navigating the challenge of using real-time analytics with limited resources and infrastructure and dealing with the need for timely insights.

Vendor selection

• Large Businesses: Evaluating and selecting from myriad complex and feature-rich AI and analytics solutions by considering factors such as scalability, integration capabilities and available vendor support.

• Midmarket: Choosing solutions that are cost-effective, user-friendly and aligned with a business’s scale and needs and making informed decisions with limited resources.

The 2023 study has also identified specific challenges that enterprises face in implementing data science, data engineering and data management services. Some of the key aspects are highlighted below.

Challenges related to data science services:

• Interpretation: Data science models, especially the ones based on complex algorithms such as deep learning may make interpretation a challenge. Enterprises continue to face difficulties in understanding and justifying the decisions made by these models — a concern in regulated industries where transparency is critical.

• Governance and Ethical Considerations: Enterprises struggle to address governance and ethical  considerations related to data science. This includes compliance with data protection regulations, ethical use of data and mitigation of biases or discrimination in algorithms — concerns factors that require proper governance frameworks.

Enterprises investing in data modernization and data engineering services are faced with the following challenges:

• Data Integration and ETL Processes: Enterprises continue to grapple with the challenge of integrating diverse data sources and undertaking extract, transform and load (ETL) processes. With the increasing
volume and variety of data, organizations need to ensure seamless data ingestion, transformation and consolidation. This includes addressing issues related to data quality, schema mapping, data compatibility
and handling real-time or streaming data.

• Scalability and Performance: In 2023, enterprises face the challenge of scaling their data engineering infrastructure to handle the growing volume of data. With continued data explosion, organizations must ensure that their systems have the ability to handle the increased workload efficiently. This involves designing scalable architectures, optimizing data processing pipelines, leveraging distributed computing frameworks and utilizing cloud-based technologies for elastic scalability.

• Real-time Data Processing: The demand for real-time analytics and insights continues to rise in 2023. Enterprises face challenges in processing and analyzing streaming data in real-time for timely decision-making and action. Building efficient real-time data pipelines, leveraging technologies such as Apache Kafka or Apache Flink, and implementing event-driven architectures are important considerations for data
engineering in the era of real-time analytics.

• Automation and Orchestration: Automating data engineering processes and orchestrating workflows are vital to improve efficiency and reduce manual efforts. In 2023, enterprises have to deal with the challenge of streamlining data engineering tasks such as scheduling data pipelines, managing dependencies and
automating data quality checks. Adopting workflow management tools, leveraging data orchestration frameworks and implementing DevOps practices are relevant in addressing this challenge.

• Cloud Migration and Hybrid Environments: As an increasing number of organizations adopt cloud technologies, enterprises face the challenge of migrating their data infrastructure to the cloud while managing
hybrid environments. They must address issues related to data integration, data movement across on-premises and cloud systems, optimizing costs and leveraging the benefits of cloud services for data
engineering workloads.

Challenges related to data management and data governance services:

• Data Quality and Master Data Management: Ensuring data quality and managing master data  effectively are ongoing challenges for enterprises. In 2023, organizations must address issues related to data consistency, accuracy, completeness and timeliness. Establishing data quality frameworks, implementing data profiling and cleansing processes, and employing master data management strategies are crucial for reliable data engineering practices.

• Data Lineage and Metadata Management: Understanding the origin and lineage of data, as well as managing metadata, is essential in 2023. Enterprises face challenges in documenting and tracking the flow of data across various systems and processes. Implementing data lineage tracking mechanisms, capturing metadata information and maintaining data catalogs or metadata repositories are key considerations for
effective data engineering practices.

• Data Collaboration and DataOps: Collaboration among different teams working with data such as data scientists, analysts and business users is crucial for success. In 2023, organizations must address the challenge of fostering collaboration, enabling self-service data access and implementing DataOps practices. This includes creating data catalogs, providing data discovery platforms and facilitating seamless  collaboration and knowledge sharing.

The absence of specific federal legal frameworks on AI in the U.S. has led to legislative and agency efforts at both federal and state levels to regulate the use of AI. These are detailed below:

• American Data Privacy and Protection Act (ADPPA):

The ADPPA is one of the proposed laws addressing AI regulation. While it primarily focuses on data privacy and protection, it includes provisions on the use of algorithms.

The bill requires impact assessments for algorithms used in decision-making that represent an elevated risk to individuals. The act reflects a growing concern about the potential negative impacts of AI-based systems on individuals and the need for accountability.

• Algorithmic Accountability Act of 2022 (AAA):

The AAA is another proposed law that specifically addresses algorithmic accountability. It emphasizes the  need for assessing their impact, particularly the ones used in decision-making processes that can significantly affect individuals.

This legislative effort underscores the recognition that algorithms, including the ones powered by AI, can have profound consequences and their deployment should be subject to scrutiny and accountability.

• State-level Consumer Privacy Laws:

In the absence of comprehensive federal legislation, individual states have taken steps to enact consumer privacy laws that regulate the collection, use and disclosure of personal data.

These state laws often include provisions related to automated decision-making, explicitly addressing AI systems in contexts such as housing, credit, employment and criminal justice. They focus on ensuring
fairness and transparency in the use of algorithms for critical decisions that impact individuals.

Both the ADPPA and AAA emphasize the need for impact assessments, signaling a shift toward proactive measures to evaluate and mitigate the potential risks associated with AI-based systems. The focus on impact assessments aligns with broader global discussions on responsible AI deployment, transparency and ethical considerations surrounding algorithmic decision-making.

While the U.S does not yet have a comprehensive federal legal framework specific to AI regulation, the legislative efforts at federal and state levels indicate a growing recognition of the importance of addressing the challenges and risks associated with, particularly related to decision-making.

Notes on quadrant positioning: In this study, several data analytics service providers that offer similar portfolio attractiveness in most quadrants have been assessed. This reflects the relative maturity of the market, providers and offerings. It is a given that not all are equal in circumstances. The vertical axis positioning in each quadrant reflects ISG’s analysis of how well the offerings align with the full scope of enterprise needs. The market has also been segmented into Large, Mid-market and Specialists to showcase the varying analytics requirements of enterprises based on their size, scale and industry dynamics. It also reflects providers’ strategy to align their portfolio and offerings to suit market demands and enterprise needs.

Access to the full report requires a subscription to ISG Research. Please contact us for subscription inquiries.