G
HÖ
KARLSKRONA
RONNEBY
AN
AN
HÖ
SKO
L
L
G
SKO
KARLSKRONA
RONNEBY
What is Availability?
Operating systems
❐
The relation between the time the system is not working and the total
time.
❐
When is the system not working?
High Availability
❐
❐
Institutionen för Programvaruteknik
och Datavetenskap
❐
Availability is not affected if the system is "down"
2.00-4.00 when the users are in their offices 8.00-17.00.
❐
Webservices usually demand 7*24*365
Operating systems: High Availability
HÖ
SKO
AN
KARLSKRONA
RONNEBY
Sida 2/32
L
G
SKO
AN
HÖ
When is when?
Sida 1/32
L
G
Operating systems: High Availability
"If a user cannot get his/her job done on time, the system is
down"
KARLSKRONA
RONNEBY
How to measure Availability
Common measure of Availability
MTBF
Time in service
Availability = --------------------------------------- = ------------------------------------MTBF + MTTR
Total time
❐
Availability figures are commonly stated as the "number of 9s"
Level Availability Downtime / year
❐
MTBF = Mean time between failure
❐
MTTR = Mean time to repair
❐
Standard deviation is important!
❐
Harddisk vendors like to state MTBF figures, but seldom the
standard deviation
❐
[9.99, 10.01, 9.999, 10.002, 9.9999] -> Mean = 10
❐
[1, 1, 1, 1, 46] -> Mean = 10
Operating systems: High Availability
Sida 3/32
1
2
3
4
5
6
90 %
99 %
99.9 %
99.99 %
99.999 %
99.9999 %
1 month
3.65 days
8.75 hours
52 minutes
5.25 minutes
31.5 seconds
Areas of use
PC
Well maintained server
Telecom
Operating systems: High Availability
Sida 4/32
G
❐
Basic System, backup data.
❐
Redundant data, hardware and/or software RAID.
❐
System failover, cluster.
10 % - Hardware.
❐
5 % - Environment.
AN
G
AN
HÖ
SKO
KARLSKRONA
RONNEBY
What to target
Downtime costs
❐
Focus on components that:
❐
Lost production -> overtime, delays.
❐
Fail the most often (low MTBF).
❐
Lost customers -> bad reputation more lost customers.
❐
Are the hardest to replace (high MTTR).
❐
Have the greatest impact when they fail.
Availability costs
❐
Sida 6/32
L
G
SKO
L
HÖ
Operating systems: High Availability
Why Availability?
❐
Power supply, fire, cable cut, dust.
Sida 5/32
KARLSKRONA
RONNEBY
❐
Careless misstakes.
❐
❐
Operating systems: High Availability
Add hardware, upgrade OS / software, preventive reboot.
15 % - People.
❐
Complete backup system at remote site.
30% server, 5 % client, 5% network.
30 % - Planned downtime.
❐
Disaster Recovery - Protect the organisation.
❐
40 % - Software failure.
❐
High Availability - Protect the system.
❐
❐
Causes for downtime
Increased Availability - Protect the data.
❐
❐
KARLSKRONA
RONNEBY
Availability index
Regular Availability - Do nothing special.
❐
❐
HÖ
HÖ
❐
AN
AN
KARLSKRONA
RONNEBY
SKO
L
L
G
SKO
About 10 times more per "9" added.
Operating systems: High Availability
Sida 7/32
Operating systems: High Availability
Sida 8/32
G
HÖ
KARLSKRONA
RONNEBY
AN
AN
HÖ
SKO
L
L
G
SKO
KARLSKRONA
RONNEBY
20 Design Principles
❐
❐
20 Design Principles
#1 Keep it simple.
❐
❐
Hard to have control of complex environment as it is
(networks, disks, applications, ...)
❐
Eliminate unneccessary hardware.
❐
Slim servers down, run only critical applications.
❐
Remove ambiguity (Who has authority to take network down?)
❐
#2 One problem, one solution.
❐
Often complex problems.
❐
Examine and fix each subproblem.
Do not invent the wheel again.
❐
Consult vendors on the implementation of your site.
#4 Reuse configurations.
❐
Easier to support, less to learn.
❐
Well tested by use.
#5 Select reliable and serviceable hardware
❐
How often will the it fail?
❐
How easy is it to fix once it fails?
Operating systems: High Availability
HÖ
KARLSKRONA
RONNEBY
20 Design Principles
❐
❐
20 Design Principles
#6 Choose mature software.
❐
#9 Invest in failure isolation.
❐
Tested by others, probably less bugs.
❐
Keeps failures from spreading.
❐
Better support.
❐
e.g. RAID can handle failure of a disk, so operation of the
database is not interrupted.
#7 Build for growth.
❐
❐
SKO
AN
KARLSKRONA
RONNEBY
Sida 10/32
L
G
SKO
AN
HÖ
❐
Sida 9/32
L
G
Operating systems: High Availability
❐
#3 Exploit external resources.
Avoid downtime for hardware upgrades.
#8 Examine the history of the system.
❐
What causes downtime.
Operating systems: High Availability
Sida 11/32
Operating systems: High Availability
Sida 12/32
G
HÖ
KARLSKRONA
RONNEBY
AN
AN
HÖ
SKO
L
L
G
SKO
KARLSKRONA
RONNEBY
20 Design Principles
❐
20 Design Principles
#10 Maintain separate environments. (Examples)
❐
❐
Production. Controlled changes.
❐
Production mirror. Controlled changes, update after the production update is proven to work.
❐
Quality Assurance. Test environment, treat it as a
production environment, but failures are not fatal.
❐
Development.
❐
Laboratory. Place to learn new technology.
❐
Disaster recovery. Production environment on another site.
❐
When is the best time to install the new compiler? (Last week
before project deadlines? )
Sida 13/32
Operating systems: High Availability
SKO
AN
HÖ
Sida 14/32
L
G
SKO
KARLSKRONA
RONNEBY
KARLSKRONA
RONNEBY
20 Design Principles
❐
What if this disk fail? What if we unplugg this network
connection?
#12 Plan ahead.
❐
AN
HÖ
❐
L
G
Operating systems: High Availability
#11 Test everything.
20 Design Principles
#13 Establish Service Level Agreements (SLA)
❐
❐
So users know what to expect. A good SLA will get the users to
work with you instead of against you.
❐
What percentage of the time is the system up? During which
hours are the system actually critical?
❐
What systems get priority?
❐
What to do if SLA is not fullfilled?
❐
Do not agree to a SLA that can not be fullfilled, think about issues that are out of control, e.g. Do not promise to have a failed
disk replaced in one day, when you do not know the delivery
time of the disk vendor (or keep one in spare).
Operating systems: High Availability
❐
Sida 15/32
#14 Document everything.
❐
History for new administrators.
❐
As memory, e.g. How did I install that software?
❐
Management can be shown what has been done.
❐
Keep documentation on paper as well!!!
#15 Automate common tasks
❐
Saves time.
❐
Less error prone.
Operating systems: High Availability
Sida 16/32
G
HÖ
KARLSKRONA
RONNEBY
AN
AN
HÖ
SKO
L
L
G
SKO
KARLSKRONA
RONNEBY
20 Design Principles
❐
20 Design Principles
#16 Consolidate your servers.
❐
Less servers to monitor.
❐
Expensive Availability solutions can be applied.
❐
Simpler environment.
❐
#17 Maintain tight security.
❐
#18 Remove Single Points Of Failures (SPOFs)
❐
RAID, Cluster.
❐
Muliple ISPs.
❐
Uninterrupted Power Supply (UPS).
Get the most bang for the buck.
Sida 17/32
Operating systems: High Availability
SKO
AN
HÖ
Sida 18/32
L
G
SKO
KARLSKRONA
RONNEBY
Data Management, Disk storage crucial
Data Management, Disk storage crucial
Disks are the most likely component to fail.
❐
Just because one component worked in one environment does
not mean it works in this.
#20 Spend money... but not blindly
❐
AN
HÖ
❐
KARLSKRONA
RONNEBY
❐
#19 Assume nothing
❐
L
G
Operating systems: High Availability
❐
❐
Approximation of time to first failure
❐
MTBF
Time to first component failure = -------------------------------------------------------Number of components
❐
Disk with 200 000 hours MTBF / 100 disks = 2000 hours to first
failure.
❐
Power supply with 30 000 hours MTBF / 6 supplies = 5000
hours to first failure.
❐
The probability that one out of many components should fail is
higher than one out of few.
Sida 19/32
Single most important asset.
The data must be protected.
❐
❐
Operating systems: High Availability
Disks contain data
Hardware can be replaced, data is harder if possible at all to replace (recreate).
Operating systems: High Availability
Sida 20/32
G
HÖ
KARLSKRONA
RONNEBY
AN
AN
HÖ
SKO
L
L
G
SKO
KARLSKRONA
RONNEBY
Disk terms
❐
❐
Disk Array - Many (or slots for) disks in one cabinet.
❐
JBOD (Just a Bunch Of Disks).
❐
RAID.
❐
❐
Operating systems: High Availability
SKO
AN
AN
HÖ
Sida 22/32
L
G
SKO
L
G
Reserve, ready to kick in when necessary.
Sida 21/32
KARLSKRONA
RONNEBY
Disk terms
❐
Add / remove disk from cabinet downtime or impact on a few
of the other disks while system is running.
Hot spare.
❐
KARLSKRONA
RONNEBY
❐
Add / remove disk from cabinet without downtime or
impact on the other disks while system is running.
Warm pluggable disk.
❐
More than one controller.
Operating systems: High Availability
HÖ
❐
Only one host can be active in accessing the data.
Multipath - More than one cable to disk (array).
❐
Hot pluggable disk.
❐
Multihost - Disk (array) is physically connected to more than one
host.
❐
❐
Disk terms
Disk terms
Write cache
❐
RAID (Redundant Array of Independent Disks)
❐
Hardware to buffer writes to disk.
❐
Software or / and Hardware.
❐
Increased performance.
❐
❐
Need proper battery backup in order not to loose data when
power fails.
Raid 0 - Stripping. write operations spread on multiple disks, no
redundancy, availability worse than single disk (more components that can fail).
❐
Raid 1 - Mirroring. Everything is stored on more than two disk.
Backup still necessary, does not protect against
removal of files.
❐
Raid 5 - Parity. Redundancy with few extra disks. If one disk
fail, that data can be recalculated from the other disks.
Storage Area Network (SAN)
❐
Storage pool that hosts can access.
❐
Centralized management and allocation.
❐
Availability easier to address, one place to improve.
Operating systems: High Availability
Sida 23/32
Operating systems: High Availability
Sida 24/32
G
HÖ
HÖ
AN
❐
AN
❐
SKO
L
L
G
SKO
KARLSKRONA
RONNEBY
KARLSKRONA
RONNEBY
Failover Requirements (for Clusters)
Failover Requirements (for Clusters)
Servers
❐
Disks
❐
Primary and Secondary.
❐
Internal, unshared disks. Used for OS and applications.
❐
Configured to the largest extent in the same way.
❐
Shared (multihost) - For critical application data.
❐
When primary fails, service is fails over to secondary.
❐
Shared nothing - Data is replicated (over the
interconnect) between the servers. Must als be used in a shared
environment if more than one server wants
access to the data.
Network connections
❐
Interconnect, between primary and secondary, used by heartbeat protocoll.
❐
Public.
❐
Administrative.
❐
Licensing issues.
No SPOFs!
Sida 25/32
Operating systems: High Availability
HÖ
SKO
AN
AN
KARLSKRONA
RONNEBY
Sida 26/32
L
G
SKO
L
G
Application portability
❐
Operating systems: High Availability
HÖ
❐
KARLSKRONA
RONNEBY
More on heartbeats
More on heartbeats
❐
Servers ping each other regularly. If a couple of heartbeats are
missed, that server can be failing.
❐
Why is a heartbeat missed?
❐
Server is down.
❐
Network interface card (NIC) used for the heartbeat has failed
-> use two NICs.
❐
❐
Heartbeat cable is unplugged / broken -> use two.
❐
Network storm on interconnect -> use two, allow for a couple
of missed heartbeats, use separate network.
❐
Network hub / switch has failed -> use two.
Operating systems: High Availability
Sida 27/32
❐
Heartbeat process has failed -> implement heartbeat in OS.
❐
Server runs too slow (no time to send heartbeat) ->
implement heartbeat in OS.
Do not under any circumstance want a heartbeat to be missed for any
other reason than that server is down.
❐
Split-brain syndrome. Both servers comes to the conclusion
(missed heartbeats) that the other is down and
continue / take over the operation -> data corruption!!!
Operating systems: High Availability
Sida 28/32
G
HÖ
KARLSKRONA
RONNEBY
AN
AN
HÖ
SKO
L
L
G
SKO
KARLSKRONA
RONNEBY
Cluster configurations
❐
❐
❐
Disaster Recovery
Asymmetric
❐
Primary doing all the work.
❐
Secondary idle, waiting to take over.
❐
Backup server(s) on different site.
❐
Separate resources
❐
Symmetric
❐
❐
The servers acts as primary and secondary for each other
❐
Both run critical applications.
❐
Each server must be capable of accepting the load of the other.
Should they be duplicated as well?
Complex return to normal operations.
N to 1 asymmetric - One machine acts as secondary for a number of
primaries.
Operating systems: High Availability
HÖ
SKO
AN
KARLSKRONA
RONNEBY
Sida 30/32
L
G
SKO
AN
HÖ
Sida 29/32
L
G
Operating systems: High Availability
KARLSKRONA
RONNEBY
Disaster Recovery
❐
Clients might be affected too.
❐
❐
Can not share resources with main site, when they are needed
they might not be there (disaster, remeber).
Final words of wishdom
Do you need disaster recovery?
❐
Monitor your resources - SNMP (Simple network management protocol).
❐
Single system failure -> no use cluster.
❐
Power failure -> use UPS and generator.
❐
Do not store backup tapes in same place as original data.
❐
Flood, fire, tornado, earthquake -> yes.
❐
Beware of SPOFs
❐
War / terrorist attack -> Yes in another country / hidden.
❐
Atomic bomb -> Probably not (why bother, nobody that can use
the system anyway).
Operating systems: High Availability
Sida 31/32
Operating systems: High Availability
Sida 32/32