DBTA Article: Overcoming the Chaos of Data Connectivity in Government
Much like organizations in every sector, federal government agencies run on data. The universal challenge of dealing with massive amounts of data every day, while leveraging the right data at the right time, is also very much front and center for government agencies.
Amid the growth of computing platforms, mobile devices, and cloud environments, government agencies have steadily added new systems and capabilities, but they’ve often integrated those systems on an ad hoc basis—as point solutions to address specific needs. Meanwhile, these agencies still rely heavily on legacy systems that are costly and time-consuming to maintain.
The result is that some data an agency needs may be tucked away in silos, making it difficult and tedious to access, or kept in formats that don’t work with modern systems. Meanwhile, agencies are on the cusp of integrating powerful new technology such as generative AI, which can add a lot of insights to help in real-time decision making when there is a need for additional data to work with.
Modernization efforts, including the deployment of Integration Platform as a Service (iPaaS), fall short in fully addressing the problem. Agencies need a platform that unifies integrations, unlocking data held throughout the enterprise, from mainframes to the edge. An emerging concept in data management called Super iPaaS could be the unified solution that puts data right at the center of mission success for government agencies.
Applications and Data Are Growing Exponentially
In government, applications and data sources are rapidly growing, which presents agencies with potentially overwhelming integration challenges. For government and the commercial sector, the number of SaaS applications alone has exploded within a 7-year period. For example, the number of SaaS applications used by organizations has grown from eight in 2015 to 130 in 2022. Overall, the SaaS industry has increased in size by 500% during this time period.
A survey by Matillion and IDG also found organizations are handling hundreds or even thousands of data sources, combining computers, mobile devices, IoT devices, websites, and other platforms. This survey found that 400 is the mean number of data sources per organization, and more than 1,000 data sources are being used by 20% of organizations. In addition, 99% of organizations will be storing data in the cloud by 2024.
Legacy Systems Persist at Government Agencies
Government agencies are also challenged by legacy systems, especially from a budgetary and procurement perspective.
Each year, the federal government spends more than $100 billion on IT and cyber-related investments. According to a recent Government Accountability Office (GAO) study, federal agencies spend $100 billion collectively to maintain legacy systems each year, which is 80% of the overall annual government IT spend.
Critically, some of those systems have aged past the point of being supported. Several years ago, the GAO reported instances in which agencies had systems with components that were at least 50 years old or vendors that were no longer providing support for outdated hardware or software.
Challenge: The Chaos of Connectivity
Government agencies have spent the better part of the last 2 decades integrating point-to-point solutions to address point-in-time problems. But as computing has evolved, with agencies continually introducing new systems and handling increasingly massive amounts of data while expanding to cloud, mobile, and edge environments, the ability to easily access large swaths of data has been lost.
The result is a “chaos of connectivity,” which is essentially a morass of systems that are integrated to an extent but don’t talk to each other. Initially, these systems were simple and fast to implement, but they have become exponentially complex across time. With the size of the agencies within the federal government, this approach is not scalable.
The mounting complexity also limits agencies’ ability to quickly access data, which may be held in siloed systems or kept in unusable formats.
One way agencies have attempted to work around this is to deploy Integration Platform as a Service (iPaaS), a cloud-based platform for managing integration between various SaaS applications.
The problem is, iPaaS often sacrifices power for ease of use or may be unable to scale mission-critical integrations across geographies, IT environments, or complex business ecosystems, resulting in multiple platforms.
Agencies need to take the next step, moving to a platform that offers a unified view of all their integration patterns, including data, applications, and APIs.
The New Concept of Super iPaaS
Super iPaaS represents a new approach to integration. Where traditional iPaaS connects SaaS applications, this new concept connects the entire enterprise.
It provides a unified interface of integrations across the organization, including hybrid multi-cloud hosting environments, geographies, user identities, and teams. Super iPaaS is managed centrally by an integration center of excellence, giving agencies quick access to data to answer requests promptly and the ability to adapt to changing technology— through a single pane of glass. This new approach offers the ability to deploy anywhere and enables the seamless flow of information, allowing agencies to react in real time to changing demands or requirements.
Through a centralized interface and real time access to data, it allows users to get more done quickly, increases productivity, and lessens the burden on the IT staff. The integrated support for APIs enables secure access to data and support for generative AI, which enables automated workflows to be created by AI models from a simple description using natural language.
As compliance is always a challenge for government, this centralized view also provides end-to-end visibility and observability across transactions throughout the enterprise.
This simplifies an agency’s ability to meet the full range of regulatory requirements.
Speeding Up Integration With Super iPaaS and AI
In addition to giving an agency full visibility and access to its integrations, Super iPaaS can help agencies save a lot of time and manual effort in creating accelerated integrations by using generative AI.
For example, when a new calendar entry is created, the AI model will send notifications—via email, text message, chat channel, or other means—to the people and resources that need to respond. AI then notices various requirements and creates a connector for the calendaring system in real time, as well as connectors to, for instance, an API or a Slack channel as needed. It also could create and implement the logic flow for that action, including mapping data formats between systems.
This provides almost 90% of what needs to be done to facilitate an integration, serving as a super accelerator. The engineer only needs to put in around 10% of their efforts to review and make minor modifications, significantly reducing their time input.
These days, agencies have more data than what can be effectively managed, and they can’t always get to the data in a seamless, efficient way. As agencies have added new systems and capabilities while also extending their enterprises into the cloud, those integrations on top of integrations have left too much of their data inaccessible.
This new paradigm goes beyond traditional iPaaS by unifying integrations throughout the enterprise, whether they involve legacy on-prem systems or applications at the far reaches of the cloud.
The visibility resulting from a unified view, along with the quick access to data regardless of where it resides or in what format, helps agencies improve performance.
Automation saves on the costs of manual operations. It also allows for the incorporation of AI models and other innovations that can make digital government a reality.
Darryn Graham is the chief architect at Software AG Government Solutions, where he provides technical and architectural thought leadership for government agencies, formulates enterprise modernization strategies, and facilitates the adoption of cloud computing, AI, and other technologies.
Read the featured article here.
