As UPMC has grown, IT officials at the $20 billion healthcare provider and insurer have found it more and more difficult to manage an increasingly sprawling network.
The Pittsburgh-based nonprofit has averaged 12% revenue growth and 7% growth in employees annually since 2001, and the organization is now the largest non-governmental employer in Pennsylvania, with 87,000 workers. What’s more, UPMC now operates 40 academic, community, and specialty hospitals, along with 700 doctors’ offices and outpatient sites.
Several years ago, the proliferation of sites started putting a strain on the healthcare system’s networking team, causing IT administrators to seek out new solutions. “We had so many sites where we were managing router configurations individually,” says Dan Snyder, principal architect for UPMC. “There could be a new doctors’ office opening every week. And when a new application rolled out, we had to change settings at hundreds of locations and coordinate between central IT teams and on-site teams. Something as simple as a quality-of-service change could take months of work by a large team of networking engineers.”
Across the healthcare sector, Snyder notes, organizations are adding new devices and applications to improve the patient experience, but IT budgets typically aren’t seeing dramatic increases. As a result, IT teams must find solutions that simplify management without substantially increasing costs.
“We were looking for something that would be easy to deploy and easy to manage at scale,” Snyder says. “I think for the entire healthcare industry, if you have a hospital and a bunch of affiliated clinician offices, you’re going to have to figure out how to manage that connectivity at all those sites without increasing costs.”
Enter Software-Defined Networking
UPMC opted to deploy software-defined networking (SDN) in its data center and connect remote sites via a software-defined wide area network (SD-WAN). The goal: both deploy and manage remote sites more quickly and easily, without negatively affecting performance or significantly increasing costs.
At first, UPMC rolled out the new infrastructure at only a handful of remote sites, to ensure that the solution would meet the organization’s needs. “We were testing out how well this worked – is it stable, does it meet our needs,” Snyder says. “It passed all of our tests for the initial scale we were looking at, and met our criteria for ease of deployment.”
Although some organizations procure SD-WAN as a managed service, UPMC opted to take advantage of its existing networking engineers to build out its own software-defined network. “If you already have a staff to manage an SD-WAN, then the as-a-service model can end up costing more money over the long run,” Snyder notes.
UPMC also took the opportunity to try using broadband Internet to connect remote sites, as a replacement for leased lines. While leased lines provide a dedicated connection with fixed bandwidth and identical upload and download speeds, broadband allows for asymmetric speeds (meaning downloads are faster than uploads) and is subject to contention with other users.
“We tried out broadband for 3-4 sites for a period of 3-6 months, and we had positive comments back from the sites,” Snyder says. “They felt that performance was better than what we were providing with leased services. We were able to go from 10 to 20 megabit download speeds to 100 megabit downloads with broadband.”
Overall, the SDN pilot was a huge success, helping UPMC to meet or exceed its goals. “The capability of the SDN product we’re putting out provides probably five to 10 times the performance as before, for about the same cost, and it’s easier for us to manage,” Snyder says.
Enabling Remote Workers
Perhaps no other use case illustrates the simplicity of UPMC’s SD-WAN solution as that of radiologists who work remotely. Like many hospitals and healthcare systems, UPMC employs a number of radiologists who work almost exclusively from their homes, meaning they can view more images and serve a larger number of hospital sites. Some radiologists live more than an hour’s drive away from the hospitals they primarily serve.
In other industries, remote workers might be able to get by with a home Internet connection, a laptop, and an assortment of cloud collaboration tools. But remote radiologists require secure connections that support fast downloads of huge digital files.
UPMC’s SD-WAN infrastructure is so simple to deploy that the organization now ships out SDN routers to radiologists’ homes, along with laminated cards that include instructions for setting up the devices. The routers come with color-coded Ethernet cables, and the radiologists themselves plug the gear into their modems and workstations. The entire process only takes a few minutes, and can be completed by healthcare professionals with no IT background. Later, if configuration changes are needed, networking staffers can make those adjustments centrally, rather than visiting every work-from-home radiologist and other remote site on the network.
“That is a huge win for managing all of the sites,” Snyder says.
In addition to the simplicity of SDN, the remote radiologists benefit from the improved performance. “It definitely makes it easier for them to go through images more quickly,” Snyder says. “They’re not waiting as long. In healthcare, the margins are really low to begin with, and you have to find ways to manage things at a larger scale without costing more money to the business.”
So far, UPMC has deployed its SD-WAN infrastructure at more than 100 remote sites, and is continuing to slowly roll out the solution across the organization’s network. Up to now, the focus has been on getting the solution up and running to simplify deployment and management. But as the environment matures, Snyder says, UPMC may try to find ways to leverage software-defined networking to better support Internet of Things (IoT) applications, improve cyber security, and redeploy networking staff to more strategic projects.
“For now, we’re able to manage more sites and a growing network, and also improve performance, without adding staff or spending significantly more money,” Snyder says. “That’s been the biggest benefit.”
UPMC continues to foster innovation and the advancement of technology for better care. To this end UPMC formed the Center for Connected Medicine (CCM), a gathering place where those seeking to drive improvements in health care through technology come to inspire each other, both in the real and digital worlds. The CCM, operated by GE Healthcare, Nokia and UPMC, connects leaders and innovators to join the CCM community by creating a relevant and authoritative content hub and fostering trusted relationships through exclusive events.