In this blog post, we’re highlighting Vodafone for the DevOps achievements that earned them the ‘Unlocking the secrets of DevOps’ award in the2022 DevOps Awards. If you want to learn more about the winners and how they used DORA metrics and practices to grow their businesses, start here.
Vodafone — a leading telecommunications company in Europe and Africa — uses technology to improve people’s lives and enable an inclusive, sustainable digital society. Since its founding in Newbury, UK in 1984, it has grown into a highly successful global company, serving millions of users across 17 different countries.
Challenges
As part of our dedication to adopting new technologies that create better customer experiences, the next phase of our growth was designed around our strategy of customers, simplicity and growth — but our past successes were actually getting in the way of the future.
The most notable ways these challenges presented themselves were:
-
Complex environment provisioning: Vodafone data scientists had to go through several manual steps, making the path from data science notebook to production environment longer and more complicated than necessary
-
Inability to scale: Since manual provisioning required customized installations for each data scientist’s Google Cloud AI deployment, it was challenging to duplicate an ML workload across each market’s unique ecosystem
-
Too little automation: Because Vodafone hosted and self-managed Jenkins on their cluster, they couldn’t take advantage of the benefits of automation
Objectives
To create our AI-driven solutions, we knew that we needed a way to unlock standardization, flexibility with clear guardrails, and high levels of automation. Whatever route we decided to take, we understood that it would need to improve scalability, end-to-end automation, value at scale, and efficiency.
Solution
To address these challenges and promote co-creation across our entities and markets, we teamed up with Google Cloud to create a new platform built on our combined technology and Agile design principles. After a commitment of eighteen months dedicated to learning and improving, the result was a new initiative called “AI Booster.” (You can read a technical description of this project here). This implementation of Vertex AIuses Google Cloud-native tools like Cloud Build, Artifact Registry, and BigQuery to drive an end-to-end ML lifecycle based on the idea of “build once, deploy many.” This method dramatically improves time-to-market while following Vodafone’s security requirements.
As the core of AI Booster, Vertex AI gives our company the power to scale, manage, monitor, and govern our ML models. The platform lets them scale AI and ML solutions both vertically from proof of concept to production and horizontally with replication across local markets. It was also built with data scientists in mind, with real-time public package and image scanning designed around DevSecOps principles.
In short, AI Booster “provides Vodafone with the ability to introduce a Global Privacy and AI governance process for AI and ML initiatives across markets” stated Mikko Niva, Global Privacy Officer for Vodafone.
With AI Booster and the benefits of the cloud, we can better comply with our 10 golden principles such as cloud native, standardized tooling, API first, always on, and reusability. The pre-built CI/CD pipeline automates every step in the environment provisioning from setup onward. And since the Vodafone Data Ocean was built on Google Cloud with on-prem hosted GitHub, a secure vault, and a mirror and orchestration system, full automation is as easy as clicking a button. This zero-touch environment reduced platform provisioning time from 3 months to 30 minutes, with teams creating over 100 instances of AI Booster across multiple markets within the first few months.
Results
Thanks to this close collaboration with Google Cloud, AI Booster now handles all new AI/ML use cases at Vodafone with plans underway to grow and accelerate adoption even further by migrating existing use cases. We have also seen significant improvements in the following areas that align to improvements in DORA four key metrics:
-
Lead times: AI Booster reduces the time it takes a user to develop a product from PoC to production from twenty weeks to two weeks, increasing efficiency by 80%.
-
Deployment frequency: Google Cloud-native tools like Cloud Build automate the pipeline and offer good reusability, eliminating 49,000 platform provisioning man-hours across Vodafone Local Markets annually and boosting ML engineer efficiency by four times.
-
Change failure rate: By automating the backend, employees can fulfill the platform setup request just by triggering a request via API, with change failure rate occurrences becoming as low as 3%.
-
Time to restore: While there haven’t been any significant service interruptions recently, it would only take involved parties about an hour to resolve the issue.
One example of AI Booster in action is Vodafone’s “Recommender,” which suggests products and services to customers based on their profile and existing services. We had to build multiple templates to overcome initial accuracy challenges, but thanks to the reduced cycle time for model deployments and re-deployments, time-to-market took 5 months for the initial market and only 6 weeks for subsequent markets. Recommender’s automated re-training pipeline kept failure rates low without having to go offline to re-train, and with automated deployment and active monitoring, restoration time went from 12 days overall to a couple hours.
As a technology team, we’re incredibly proud of our cutting-edge MLOps platform based on best-in-class Google Cloud architecture with built-in automation, scalability, and security. With AI Booster, we’re delivering more value from data science while embedding reliability engineering principles throughout the organization.
Stay tuned for the rest of the series highlighting the 2022 DevOps Award Winners and read the 2022 State of DevOps report to dive deeper into the DORA research.