NYS Forum Business Continuity Work Group Meeting

Date: Held : May 19, 2010 @ 1:30 PM
CGI offices, Corporate Woods, Albany

Discussion Topic

Attendance

  • AT&T - Al Rodriguez
  • CGI - Mark Spreitzer
  • DHCR - Richard Siek
  • DMV - Jeff Deal; Matt Kirchner; Mark Hammond; Mike Hicks; Jason Berry
  • EMC - Ed Walsh
  • IPLogic - Scott Elliott; Chris Fox
  • NY DOS - Caroline; Bob Stevens
  • NYSED - Ken Mason
  • NYS Forum - Greg Benson
  • OFT - Benita Sokolowski

Member updates

During introductions, attendees reported on business continuity and DR related activities at their agencies

  • Mark Spreitzer, CGI - mark will be attending security conferences and will bring observations back for next work group meeting
  • Caroline, DOS -- VMware environment and starting to think in terms of DR
  • Bob Stevens, DOS -- grand plans for virtualization in DR
  • Richard Siek, DHCR -- finished draft COOP; IT staff working on virtualization for DR
  • Benita Sokolowski, OFT -- implementing NY Alert, toward the end of it; starting a BIA on back-end to validate mission-essential
  • Jeff Deal, DMV -- presenting DMV's story on VM; started a COOP process
  • Ed Walsh, EMC -- virtualization practice at EMC
  • Chris Fox, IPLogic -- a lot of interest in DR with VMware, inefficiencies around traditional DR
  • Al Rodriguez, AT&T -- new member, interested in needs of agencies
  • Matt Kirchner, NYS DMV -- technical staff, SAN support
  • Mark Hammond, NYS DMV -- there are facility constraints associated with DR
  • Mike Hicks, NYS DMV -- get the network in two locations at once, securely and resiliently
  • Jason Berry, NYS DMV -- SAN & data archiving

NYS Forum Updates

  • Mark commented on upcoming consolidated task group event hosted by the PM work group that will discuss BC and Security in the System Development Lifecycle, (SDLC) as an update to the NYS PM Guidebook. Registration on NYS Forum site. 1 week from today at 9:00 .a.m. at Desmond.
  • Consolidated Task Group for Web 2.0 developed a Facebook page for the NYS Forum as a means to connect with fellow forum members. An event is planned for June 22, 2010 to talk about social networking and government adoption.
  • Mark will be speaking at World Conference of Disaster Management (WCDM) in June and the Canadian Government Security Conference (CADSI/CANSEC) in June.

Virtualization Panel Discussion

  • Jeff Deal, NYS DMV
  • Ed Walsh, EMC
  • Chris Fox, IPLogic
  • C Fox:
    • addressed common causes of downtime;
    • RPO/RTO
    • testing is important and virtualization allows for more flexible testing;
    • traditional DR involves complex infrastructure requirements and thorough training of staff, dedicated hardware
    • recovery often takes hours or days
    • testing traditional DR isn't always comprehensive
  • E Walsh:
    • addressed common uses of virtualization
    • observed that NYS is moving more and more critical services to electronic format (e.g., mybenefits, e-licensing)
    • there are many prevalent virtualization technologies in NYS (e.g., VMware, Solaris Containers, IBM LPARS, HyperV, etc)
    • Key technologies/benefits in DR: reduce HW requirements at DR sites, data deduplication to reduce storage and WAN requirements, often translates into reduced cost
    • Virtualization for DR is a top priority according to analyst reports;
    • Virtualizing allows you to protect data by moving it to other locations quickly via Backup/Recovery method or Deduplication/Replication method.
    • Most disasters are not an entire site going away, so there is much overlap with BC.
    • Roughly, 5% of data is generated new each day, the rest is duplicate data.
    • Replication (e.g., snapshots) of production virtual machines is available today. This also allows for DR tests without taking production systems offline.
    • Infrequent testing introduces problems due to system changes.
    • Automation of recovery is a big advantage of virtualization--script and make recovery repeatable.
    • It's important to make sure that replication technology is supported by/integrated with your virtual environment.
  • J Deal:
    • DMV is in the process of virtualizing the majority of its environment and is leveraging this virtualized environment to provide a level of DR that is not cost effective in a physical environment
    • In 2005 they had a complex business requirement involving need for 4 interconnected servers. VMware was used and the next day it was working.
    • Quick creation and deployment of VMs made it very attractive.
    • Started an aggressive P2V migration and saw many benefits. Tape remained primary backup.
    • There are many tools to allow recovery directly from tape to VM.
    • Virtualization enabled satisfaction of extreme pent-up demand for servers and load-balancing.
    • 120 obsolete physical servers were virtualized into 3 Hosts.
    • Started asking, "why can't we virtualize this server?"
    • Goal: every mission critical app was to be virtualized. "DMV in a box."
    • Biggest constraint became the SAN due to explosion of VMs; this created a backup 'crisis' with tape drives.
    • Archive and data retention policies had to be considered. In physical model, ran multiple services on each machine.
    • Used multiple appliances to enable SAN, NAS, Deduplication, Backup and Archiving solutions
    • Developed RTO/RPO in IT, not at business units and that need to be revisited.
    • Current state: The virtualized environment and combined with the SAN, NAS, Deduplication, Backup and Archiving improvements have enabled DMV to maximize limited DR facility space.
    • In addition, a POC of automated DR recovery is currently running and it's showing great promise.
    • Network connectivity is a challenge; LAN/WAN, DMZ, Internet, OFT connectivity.
    • Virtualization has given DMV a viable path to a more efficient and effective DR.

Panel Questions

  • Question: how often are organizations doing traditional DR?
    • E Walsh--usually a subset of applications, not entire environments
  • Question: are many organizations keeping hot sites at 1/2 capacity rather than a cold DR site?
    • E Walsh--usually for BC planning, yes, if the sites are close by. More frequently DR site runs test/dev and main site runs production.
    • C Fox--test/dev at secondary site is most common.
    • E Walsh--related topic: splitting between public/private clouds is of great interest, though it is not fully mature yet.
  • Question: do you need to worry about which applications are on which virtual server?
    • C Fox: not too much of a concern; what works in production will work in DR and the automated recovery can be scripted to be back online quickly
  • Question: if I'm starting down the path of virtualization, what are the key triggers to help me decide which apps to start with?
    • C Fox: low hanging fruit is often the first thing done (e.g., file & print), but database and email applications are the best targets as critical infrastructure. Mark Hammond: physical servers are just not cost effective anymore; ultimately virtual to virtual recovery is the goal
  • Question: has anyone done $ calculations to ROI of virtualization?
    • Mark Hammond: IBM did a calculation for DMV and Jeff Deal will cover in his preso.
  • Comment: Ewalsh--cloud apps that are adopted need to be integrated into the recovery plan if there are application connections to them.
  • Question: how is security tracked in these virtual environments?
    • Cfox: there are products that enforce security policies (e.g., firewall rules) in virtual environments?
    • E Walsh: virtual environments are in some ways easier to secure than physical since all server characteristics exist within a file; network infrastructure is being virtualized in parallel with server
  • Question: any network gotchas?
    • J Deal: network was a concern early on; large servers with a lot of NICs. Moved to blade environment with 6 NICs per blade and are migrating the environment to blade. Memory is the constraint.
  • Comment: Do not let vendor white papers and data sheets dictate what is possible; challenge it.

Action Items

Action Item
Rolling List of Possible Session Topics
  • Infrastructure recovery of DR and how virtualization can play a role in helping to reduce cost, offered as a panel discussion with a few vendors
  • Auditor Panel Discussion
  • Partner with the Emerging Technologies work group to request a representative from Web 2.0 Consolidated Task Group discuss with us social media usage in disaster response
Owner
Mark Spreitzer
Due Date
N/A
Action Item
Represent BC Work Group at Strategic Planning Meeting
Owner
Ken Mason
Due Date
July 16
Action Item
Work with Rebecca and Greg regarding scheduling a date that would work for the Gartner opportunity and posting a call for speakers to support NYS Forum process
Owner
Ken Mason
Due Date
September

Presentation

Note: Presentation link below open in a new window. If you are unable to open the provided format or require a different one, please contact us and we will provide an alternative format.

Next Meeting: June 23, 2010, 1:30 PM to 3:30PM at CGI offices in Corporate Woods