On July 18, shortly before midnight, a routine software update turned into the largest information technology outage in history, affecting nearly 8.5 million Windows devices worldwide.
Critical systems, from hospitals and airports to banks, government agencies, and emergency services, were suddenly brought to a halt. The incident “gave millions of computers running Windows and CrowdStrike the blue screen of death,” says Tiffany Hanulec, executive director of technology services & chief technology officer in Penn’s Information Systems & Computing Offices (ISC).
Hanulec says IT teams across the University acted quickly, deducing the incident was caused by the latest software update by the company CrowdStrike, and shared knowledge to repair the issue. ISC’s teams worked into the night to manually roll back the faulty update on more than 600 computer servers, coordinating with IT staff around campus.
“Fortunately, Penn’s teams were well positioned to recover quickly, relative to larger entities like Delta Airlines, which took several days to bounce back.” Hanulec says.
Hanulec and Jaron Rhodes, ISC director of communications, attribute ISC’s success in figuring out the CrowdStrike issue to the “wait and see” approach they take to new software and technology rollouts. Rhodes says, “Instead of rushing into the latest tech, our teams assess its stability first, ensuring our systems remain secure and resilient.”
ISC’s response to shutdowns like those seen in the CrowdStrike incident is just one of the myriad aspects of work by the growing ranks of IT professionals at Penn. Ongoing maintenance is critically important, while also keeping up with new technologies and trends that pose security risks.
The IT crowd at Penn
Penn’s IT workforce has expanded by about 20% in the past decade to about 1,250 professionals, as technology becomes increasingly integral to the University’s operations.
“The expectations for IT and the demand we see on campus are always growing,” Hanulec says, especially following the pandemic, which accelerated the adoption of remote and hybrid tools.
Hanulec explains that ISC serves as Penn’s central IT organization, managing essential services like the campus network, email systems, and enterprise applications, offering what they call “common good” services that benefit the entire University. Local Support Providers (LSPs) in individual schools and centers handle individual technology needs.
Planning for advance technology
Much of the work of the ISC teams relates to keeping the systems running smoothly, such as maintenance and steady expansion of IT support infrastructure to match the cadence of rapidly advancing technology and increased adoption of these tools by Penn’s users, Hanulec says.
To meet the growing demands of an expanding user base, ISC and campus partners created Penn’s new IT Strategic Plan, focusing on student experience, academic technology, research computing, and data and AI. This work was done in alignment with Penn’s strategic framework, In Principle and Practice.
“We stand at the brink of another transformative technological revolution, reminiscent of the emergence of the internet and personal computers,” says Tom Murphy, senior vice president for information technology & University chief information officer. “To position Penn for this future, ISC has partnered with campus stakeholders to create a new IT Strategic Plan, ensuring that the University continues to lead in innovation and excellence.”
ISC is focused on increasing efficiency and security in administrative centers through Penn SecureIT. Hanulec says this project allows staff to concentrate on tasks that have a broader impact by freeing them from routine IT maintenance.
“Information security is an emergent property of IT done well, and IT done well is largely IT done at scale,” says Nick Falcone, executive director of information security and University information security officer.
Another significant project is the Identity & Access Management initiative, which is enhancing security by implementing advanced authentication systems. “We’re moving towards more secure, user-friendly methods of verifying identity, such as advanced multifactor authentication,” Rhodes says.
Hanulec highlights how the increasing use of AI tools like ChatGPT poses both opportunities and concerns.
“We’re seeing these AI tools being adopted more frequently for basic tasks, decision-making, and even generating project names,” she says. However, she cautions that “not all AI tools are created equally” and explains that AI tools can sometimes provide misleading or incorrect information. “You should always fact-check,” she says.
Another pressing concern is the privacy and security implications of using AI tools that record meetings or collect data without proper authorization. Hanulec poses the example of third-party AI tools that might record meetings without consent, inadvertently breaking privacy laws and sharing information.
“People need to think through what it means when using these AI tools,” she says, “especially if the tools are storing sensitive Penn data on external servers,” as this raises concerns about where that data might end up and how secure it really is, a significant issue for privacy and security teams.
To address these challenges, ISC is working closely with the Office of Audit, Compliance, and Privacy to vet AI tools before they’re made available on campus, and together they produced guidance on safe AI tools.
