Half hour to ghost the server, then patch. If there is a problem, it's
another half hour to fix it (sometimes less).
I have baseline images of all my servers that are periodically refreshed.
Then I have a 'current' image branch that is taken before patching. This is
all stored on a terabyte RAID-5 array.
This way, if there is ever a serious disaster, I can have the baseline image
restored in under 45 minutes. I can pull the most current dynamic
information off of the daily tape backups (things like mailboxes, databases,
home directories, etc).
All in all, restore time for my SQL server, for example, goes from seven
hours with tape, to about an hour total with Ghost and tape. The latest rev
of ghost even has a backup system, so theoretically I could schedule
incremental images to be taken, but I haven't set that up yet.
The obvious downside to Ghost is downtime. We aren't a 24/7 operation, so I
can take the servers down for maintenance, patching, and testing at night,
but that might be unacceptable for another operation. The other question is
backups of your backups. My images, with compression, sometimes run as high
as 25gb (DB server), hence the terabyte array.
I'm on a tight budget, working for a non-profit, so duplicating hardware is
out. VMWare, some spare desktop machines and Ghost are the answer for us.
Our main concern is our Metaframe farm and the custom apps we publish
through it. For software conflicts, we test out in our virtual test lab. For
hardware conflicts (Anyone remember the NT 4.0 Security Rollup and Compaq
Array controllers? Been there, done that.) we use Ghost.
The main patch testing problem I run into is simulating a multi-user
environment. A Metaframe server, sitting in a corner, with two or three
people logged onto it might behave fine.
Then you go to production, where you have a hundred people logging on and
off at any given time, and the whole game plan changes. I run resource
starvation benchmarks on the test servers and the like, but that just isn't
quite the same as a hundred users doing things and opening and closing
things. *That* is the hardest part to emulate, IMHO. If you have the exact
same hardware and software, you still don't have the exact same users.
Best Regards,
Ryan Cronk - IT Engineering
Columbia Legal Services
-----Original Message-----
From: Merriman, Jason [mailto:jmerriman (at) above (dot) net [email concealed]]
Sent: Tuesday, August 26, 2003 10:17 AM
To: focus-ms (at) securityfocus (dot) com [email concealed]
Subject: RE: Patch testing
If you don't have mirrored disk capabilities - use Norton Ghost to snap an
image of the system partition on the server before patching. Ive found that,
with compression, most images I've taken are less than 6 GB.
It's much easier to restore a Ghost image than to install an OS from scratch
and restore from tape.
--------------------------------------
Jason Merriman, Senior Site Manager
AboveNet
jmerriman (at) above (dot) net [email concealed]
cell - 703.447.8402
tel - 571.633.5166
aim - revision
--------------------------------------
> -----Original Message-----
> From: Russell V. Toone [mailto:russ (at) eCallogy (dot) com [email concealed]]
> Sent: Tuesday, August 26, 2003 11:22 AM
> To: focus-ms (at) securityfocus (dot) com [email concealed]
> Subject: RE: Patch testing
>
>
>
> Just a thought, use mirrored disks, then before installing
> the patch(es) break the mirror and install to one of the
> disks. If all goes well, set the mirroring back up, if it
> doesn't go well, then fire up the machine from the "broken"
> mirror disk, and you'll be back where you were right before
> the patch installation.
>
>
> -----Original Message-----
> From: Kurt Seifried [mailto:bt (at) seifried (dot) org [email concealed]]
> Sent: Monday, August 25, 2003 2:39 PM
> To: Matt Brei
> Cc: Todd Schubert; focus-ms (at) securityfocus (dot) com [email concealed]
> Subject: Re: Patch testing
>
> > And we're supposed to do this for every patch MS releases? I would
> > have to hire 5 guys just to test patches. Not to mention
> that if one
> > of the patches does fail we have a production server(s)
> down. Backups
> > are great yes, but have you ever done a 100+ GB restore
> from an Ext.
> > SCSI LTO drive? It takes about 14 hours.
>
> Then this is one of the additional costs of running Windows
> that you will need to accept. Or else you can forego patch
> testing, cross your fingers and hope nothing breaks. People
> who complain about this strike me as a bit odd, you did know
> about these problems going into your MS purchase, didn't you?
> TCO, blah blah blah.
>
> As far as making this mess easier to deal with there are some
> potential
> lights:
>
> VMWare Workstation- pro: it's cheap, it's easy, you can test
> many configs quickly. cons: hardware issues probably will not
> come up. VMWare GSX/ESX/etc - pro: you can segment a server
> and test patches on "identical" setups, using only one
> hardware, cons: it's not cheap.
>
> Removable harddrives (if you can afford server downtime):
> simply swap the drives, go to a "test" drive, see if it blows
> up, if not go to production drive.
>
> Ultimately if you have a server so critical that it can't go
> down for more then a few minutes, and you have no
> backup/recovery plan that accounts for a hardware failure
> (they do happen) then you are screwed equally for testing and backup.
>
> And as another poster mentioned this is likely to only get
> worse, with part of IIS 6 running in kernel mode to increase
> speed (ala Tux web server on Linux, except IIS is a whole lot
> more complicated). VPN's are increasingly going to rely on
> hardware acceleration, especially on the servers, security
> updates may affect the drivers there.
>
> > Matt Brei
> > Network Administrator
>
>
>
>
> Kurt Seifried, kurt (at) seifried (dot) org [email concealed]
> A15B BEE5 B391 B9AD B0EF
> AEB0 AD63 0B4E AD56 E574
> http://seifried.org/security/
>
>
>
>
> --------------------------------------------------------------
> -------------
> KaVaDo provides the first and only integrated Web application
> scanner and
> firewall security suite that prevent Web applications
> attacks, the most
> common form of online exploitation. Download a FREE
> whitepaper on Security Policy Automation for Web Applications.
> http://www.securityfocus.com/sponsor/KaVaDo_focus-ms_030818
> --------------------------------------------------------------
> -------------
>
> --------------------------------------------------------------
> -------------
> KaVaDo provides the first and only integrated Web application
> scanner and
> firewall security suite that prevent Web applications
> attacks, the most
> common form of online exploitation. Download a FREE
> whitepaper on Security Policy Automation for Web Applications.
> http://www.securityfocus.com/sponsor/KaVaDo_focus-ms_030818
> --------------------------------------------------------------
> -------------
>
------------------------------------------------------------------------
---
KaVaDo provides the first and only integrated Web application scanner and
firewall security suite that prevent Web applications attacks, the most
common form of online exploitation. Download a FREE whitepaper on Security
Policy Automation for Web Applications.
http://www.securityfocus.com/sponsor/KaVaDo_focus-ms_030818
------------------------------------------------------------------------
---
------------------------------------------------------------------------
---
KaVaDo provides the first and only integrated Web application scanner and
firewall security suite that prevent Web applications attacks, the most
common form of online exploitation. Download a FREE whitepaper on Security Policy Automation for Web Applications.
http://www.securityfocus.com/sponsor/KaVaDo_focus-ms_030818
------------------------------------------------------------------------
---
Half hour to ghost the server, then patch. If there is a problem, it's
another half hour to fix it (sometimes less).
I have baseline images of all my servers that are periodically refreshed.
Then I have a 'current' image branch that is taken before patching. This is
all stored on a terabyte RAID-5 array.
This way, if there is ever a serious disaster, I can have the baseline image
restored in under 45 minutes. I can pull the most current dynamic
information off of the daily tape backups (things like mailboxes, databases,
home directories, etc).
All in all, restore time for my SQL server, for example, goes from seven
hours with tape, to about an hour total with Ghost and tape. The latest rev
of ghost even has a backup system, so theoretically I could schedule
incremental images to be taken, but I haven't set that up yet.
The obvious downside to Ghost is downtime. We aren't a 24/7 operation, so I
can take the servers down for maintenance, patching, and testing at night,
but that might be unacceptable for another operation. The other question is
backups of your backups. My images, with compression, sometimes run as high
as 25gb (DB server), hence the terabyte array.
I'm on a tight budget, working for a non-profit, so duplicating hardware is
out. VMWare, some spare desktop machines and Ghost are the answer for us.
Our main concern is our Metaframe farm and the custom apps we publish
through it. For software conflicts, we test out in our virtual test lab. For
hardware conflicts (Anyone remember the NT 4.0 Security Rollup and Compaq
Array controllers? Been there, done that.) we use Ghost.
The main patch testing problem I run into is simulating a multi-user
environment. A Metaframe server, sitting in a corner, with two or three
people logged onto it might behave fine.
Then you go to production, where you have a hundred people logging on and
off at any given time, and the whole game plan changes. I run resource
starvation benchmarks on the test servers and the like, but that just isn't
quite the same as a hundred users doing things and opening and closing
things. *That* is the hardest part to emulate, IMHO. If you have the exact
same hardware and software, you still don't have the exact same users.
Best Regards,
Ryan Cronk - IT Engineering
Columbia Legal Services
-----Original Message-----
From: Merriman, Jason [mailto:jmerriman (at) above (dot) net [email concealed]]
Sent: Tuesday, August 26, 2003 10:17 AM
To: focus-ms (at) securityfocus (dot) com [email concealed]
Subject: RE: Patch testing
If you don't have mirrored disk capabilities - use Norton Ghost to snap an
image of the system partition on the server before patching. Ive found that,
with compression, most images I've taken are less than 6 GB.
It's much easier to restore a Ghost image than to install an OS from scratch
and restore from tape.
--------------------------------------
Jason Merriman, Senior Site Manager
AboveNet
jmerriman (at) above (dot) net [email concealed]
cell - 703.447.8402
tel - 571.633.5166
aim - revision
--------------------------------------
> -----Original Message-----
> From: Russell V. Toone [mailto:russ (at) eCallogy (dot) com [email concealed]]
> Sent: Tuesday, August 26, 2003 11:22 AM
> To: focus-ms (at) securityfocus (dot) com [email concealed]
> Subject: RE: Patch testing
>
>
>
> Just a thought, use mirrored disks, then before installing
> the patch(es) break the mirror and install to one of the
> disks. If all goes well, set the mirroring back up, if it
> doesn't go well, then fire up the machine from the "broken"
> mirror disk, and you'll be back where you were right before
> the patch installation.
>
>
> -----Original Message-----
> From: Kurt Seifried [mailto:bt (at) seifried (dot) org [email concealed]]
> Sent: Monday, August 25, 2003 2:39 PM
> To: Matt Brei
> Cc: Todd Schubert; focus-ms (at) securityfocus (dot) com [email concealed]
> Subject: Re: Patch testing
>
> > And we're supposed to do this for every patch MS releases? I would
> > have to hire 5 guys just to test patches. Not to mention
> that if one
> > of the patches does fail we have a production server(s)
> down. Backups
> > are great yes, but have you ever done a 100+ GB restore
> from an Ext.
> > SCSI LTO drive? It takes about 14 hours.
>
> Then this is one of the additional costs of running Windows
> that you will need to accept. Or else you can forego patch
> testing, cross your fingers and hope nothing breaks. People
> who complain about this strike me as a bit odd, you did know
> about these problems going into your MS purchase, didn't you?
> TCO, blah blah blah.
>
> As far as making this mess easier to deal with there are some
> potential
> lights:
>
> VMWare Workstation- pro: it's cheap, it's easy, you can test
> many configs quickly. cons: hardware issues probably will not
> come up. VMWare GSX/ESX/etc - pro: you can segment a server
> and test patches on "identical" setups, using only one
> hardware, cons: it's not cheap.
>
> Removable harddrives (if you can afford server downtime):
> simply swap the drives, go to a "test" drive, see if it blows
> up, if not go to production drive.
>
> Ultimately if you have a server so critical that it can't go
> down for more then a few minutes, and you have no
> backup/recovery plan that accounts for a hardware failure
> (they do happen) then you are screwed equally for testing and backup.
>
> And as another poster mentioned this is likely to only get
> worse, with part of IIS 6 running in kernel mode to increase
> speed (ala Tux web server on Linux, except IIS is a whole lot
> more complicated). VPN's are increasingly going to rely on
> hardware acceleration, especially on the servers, security
> updates may affect the drivers there.
>
> > Matt Brei
> > Network Administrator
>
>
>
>
> Kurt Seifried, kurt (at) seifried (dot) org [email concealed]
> A15B BEE5 B391 B9AD B0EF
> AEB0 AD63 0B4E AD56 E574
> http://seifried.org/security/
>
>
>
>
> --------------------------------------------------------------
> -------------
> KaVaDo provides the first and only integrated Web application
> scanner and
> firewall security suite that prevent Web applications
> attacks, the most
> common form of online exploitation. Download a FREE
> whitepaper on Security Policy Automation for Web Applications.
> http://www.securityfocus.com/sponsor/KaVaDo_focus-ms_030818
> --------------------------------------------------------------
> -------------
>
> --------------------------------------------------------------
> -------------
> KaVaDo provides the first and only integrated Web application
> scanner and
> firewall security suite that prevent Web applications
> attacks, the most
> common form of online exploitation. Download a FREE
> whitepaper on Security Policy Automation for Web Applications.
> http://www.securityfocus.com/sponsor/KaVaDo_focus-ms_030818
> --------------------------------------------------------------
> -------------
>
------------------------------------------------------------------------
---
KaVaDo provides the first and only integrated Web application scanner and
firewall security suite that prevent Web applications attacks, the most
common form of online exploitation. Download a FREE whitepaper on Security
Policy Automation for Web Applications.
http://www.securityfocus.com/sponsor/KaVaDo_focus-ms_030818
------------------------------------------------------------------------
---
------------------------------------------------------------------------
---
KaVaDo provides the first and only integrated Web application scanner and
firewall security suite that prevent Web applications attacks, the most
common form of online exploitation. Download a FREE whitepaper on Security Policy Automation for Web Applications.
http://www.securityfocus.com/sponsor/KaVaDo_focus-ms_030818
------------------------------------------------------------------------
---
[ reply ]