Link:http://www.wikisummaries.org/Visible_Ops

Contents

[hide

What is ITIL?

  • ITIL = Information Technology Infrastructure Library
  • A "drastically different approach to IT" (p79)
  • A "maturity path for IT that is not based on technology" (p79)
  • A "collection of best practices codified in seven books by the Office of Government Commerce in the U.K." (p85)
  • A collection "without prioritization or any prescriptive structure" (p18)
  • Used by Visible Ops authors as a framework "to normalize terminology" and categorize traits shared across studied high performing organizations (p18-20)

Introduction

(p10-24)

What is Visible Ops?

  • Highest ROI best practices divided into four prioritized and incremental Phases
  • All ideas are mapped to ITIL terminology
  • Intended to be an "on-ramp" to ITIL

Key premises to the Visible Ops rational

  1. 80% of unplanned outages are due to ill-planned changes made by administrators ("operations staff") or developers
  2. 80% of Mean Time To Repair (MTTR) is spent determining what changed
  3. With the right processes in place, it is easier, better, and more predictable to rebuild infrastructure than to repair it
  4. Concentrating staff time on pre-production efforts is more efficient and less expensive due to the high cost of repairing defects while in production
  5. Without process controls, pieces of infrastructure often become like unique snowflakes or irreplaceable works of art ... only understood by the "rocket scientist" creator who's time is tied to maintaining it (p41)
  6. "You can not manage what you can not measure" (p59)

Phase One: Stabilize the Patient

(p25-40)

Goals

  • Identify most critical IT systems generating the most unplanned work
  • Stabilize infrastructure (prioritizing the most fragile components)
  • Create a "culture of causality" where all changes are viewed as key risks that need to managed by facts rather than by beliefs
  • Reduce unplanned work to 25% or less (high performers achieve lower than 5%)
  • Maximize change success rates (high performers hit 98%)
  • Minimize Mean Time to Repair (MTTR)
  • Ensure security specialists become part of the decision process
  • Shift staff time from "perpetual firefighting to more proactive work that addresses the root causes of problems"
  • Minimize the IT failures that cause stress and damage IT's reputation
  • Increase the overall level of confidence in IT
  • Collect data to affirm the new processes and foster an understanding that any previous perceptions of nimbleness and speed were not factoring in time spent troubleshooting and doing unplanned work

Recommended key steps (to be implemented on most fragile systems first)

  1. Reduce or eliminate change privileges to fragile infrastructure

    • Why? Every time a change is made you risk breaking functionality
  2. Create scheduled maintenance windows where all changes are made
    • Why? Scheduled changes are more visible, and are more likely to be planned and tested before going into production
  3. Automate daily scans to detect and report changes
    • Why? To automatically verify and log that all scheduled changes were made ... and that no other changes were made
    • Warning: Due to their collected data, the authors strongly recommend that even the most trusted administrators still work under automated detections
    • Disclosure: One of the authors is the CTO at Tripwire, Inc, the manufacturer of the recommended software for these automated scans....
  4. When troubleshooting incidents, first analyze the recent changes (approved and detected) to isolate likely causes before recommending additional changes
  5. Schedule a weekly Change Advisory Board (CAB) made up of representatives from operations, networking, security and the service desk
    • Why? To ensure key stakeholders collectively inform and influence change decisions
  6. Create a Change Advisory Board - Emergency Committee (CAB/EC) who can assemble quickly to review emergency change requests
    • Why? "Emergency changes are the most critical to scrutinize"
  7. Create a Change Request Tracking System to document and track requests for changes (RFCs) through authorization, verification, and implementation processes
    • Why? To facilitate the change approval process and to generate reports with metrics

Phase Two: Catch & Release and Find Fragile Artifacts

(p41-46)

Goals

  • Prioritize IT's most critical services
  • Identify critical pieces of production infrastructure (hardware and software)
  • Identify interdependencies between components of production infrastructure
  • Foster organizational learning
  • Identify the high-risk "fragile artifacts"

Recommended key steps

  1. Create a prioritized service catalog that documents the most critical services
  2. Create a Configuration Management Database (CMDB) that illustrates mappings between services and infrastructure, and shows the interdependencies between all configuration items (CI)
  3. Freeze all related configurations for an agreed upon change-free window
    • Why? To ensure an accurate inter-related configuration inventory (see below)
  4. Inventory all equipment and software in the data center, recording the whos, whats, interdependencies and history for each item
    • Why? To facilitate faster problem management and to inform change decisions
    • Note: This inventory should be implemented by the most senior staff to ensure the most knowledgeable capturing of configuration details and histories
  5. Identify the "fragile artifacts" that have the worst historical change success rates and/or the least technical mastery by the supporting technicians, and prioritize them by the criticality of the services they provide
    • Why? To create a prioritized list of servers to rebuild in Phase Three
  6. To the extent possible, place fragile artifacts under a permanent configuration freeze until they can be replaced by complete rebuilds in Phase Three

Phase Three: Establish Repeatable Build Library

(p47-58)

Goals

  • Remove processes that encourage heroics in rewarding vigilant firefighters
  • Increase team-level technical mastery of production infrastructure
  • Shift senior staff from firefighting to fire prevention
  • Ensure that critical infrastructure can be easily rebuilt
  • Enable a new troubleshooting process with a short, predictable Mean Time To Repair (MTTR)
  • Ensure perfect configuration synchronization between pre-production and production servers
  • Ensure all configurations and build processes are completely documented

Recommended key steps (to be implemented on most fragile systems first)

  1. Create and maintain a versioned, Definitive Software Library (DSL) for all acquired and custom developed software and patches

    • Note: additions must be approved by the Change Approval Board (CAB)
    • Exception: at the time of initial creation, all currently used production software will be accepted into the DSL under a one year grace period
  2. Create a team of release management engineers from your most senior operations staff. Only more junior staff will be on the production operations team.
  3. Prevent developers and the release management engineers (previously the senior operations staff) from accessing production infrastructure
    • Reason 1: Policy encourages recommended changes to be error free with bullet-proof installation and back-out processes in place
    • Reason 2: Process verifies completeness and accuracy of documentation for installation and operations procedures
  4. Release management engineers create automated, consolidated, integrated, patched, tested, security scanned, layer-able build packages which will then be provisioned onto production infrastructure by the more junior, production operations staff
    • Reason 1: Consolidates the number of unique configuration counts (and thus increases team mastery of those fewer configurations)
    • Reason 2: Ensures fully integrated quality assurance tests and security verifications
  5. Updates and even non-emergency patches are then rolled into a new a "golden build" which is then applied to production hardware as a new build
    • Reason 1: Eliminates the risk of "patch and pray"
    • Reason 2: Otherwise, over time, break/fix cycles tend to encourage configuration variance between production and pre-production servers ... and between similar servers that should be identical
    • Reason 3: Applying new builds allows for highly accurate predictions of downtime, reduces chances of human error, and is typically faster than applying numerous individual patches and updates
  6. As a general rule, installed build packages will be preceded by erasing the production hard drive (or partition) ... the book calls this a "bare-metal build"
    • Why? This process ensure that production servers do not contain any hidden dependencies, and guarantees that the "golden builds" accurately reflect production systems, enabling perfect synchronization with pre-production servers

Phase Four: Enable Continuous Improvement

(p59-64)

Goals

  • Continuous increase in technical mastery of production infrastructure by reducing configuration variance
  • Continuous improvement of change success rates
  • Continuous increases in effective rate of change
  • Continuous monitoring to avoid slips in performance

Recommended key steps

  1. Use recommended metrics to hone efforts from the first three Phases. A few selected examples:

    • Percent of systems that match known good builds (higher is better)
    • Time to provision known good builds (lower is better)
    • Percent of builds that have security sign off (higher is better)
    • Number of authorized changes per week (higher is better)
    • Change success rate (higher is better)
  2. Strive to implement additional recommended improvement points. A few selected examples:
    • Segregate the development, test, and production systems to safeguard against any possible unintentional crossovers or hidden dependencies
    • Enforce a standard build across all similar devices
    • Define bullet-proof back out processes to recover from failed or unauthorized changes
    • Internalize the fundamental relationship between Mean Time to Repair (MTTR) and availability. By improving MTTR you also improve overall availability.
    • Track repeat offenders who circumvent change management policies.

Visible Ops的更多相关文章

  1. 关于DevOps你必须知道的11件事

    转自:http://www.infoq.com/cn/articles/11devops 关于作者 Gene Kim在多个角色上屡获殊荣:CTO.研究者和作家.他曾是Tripwire的创始人并担任了1 ...

  2. hdu2848 Visible Trees (容斥原理)

    题意: 给n*m个点(1 ≤ m, n ≤ 1e5),左下角的点为(1,1),右上角的点(n,m),一个人站在(0,0)看这些点.在一条直线上,只能看到最前面的一个点,后面的被档住看不到,求这个人能看 ...

  3. display:none与visible:hidden的区别 slideDown与

    display:none与visible:hidden的区别 display:none和visible:hidden都能把网页上某个元素隐藏起来,但两者有区别: display:none ---不为被 ...

  4. toArray(),toJson(),hidden([ ]),visible([ ])

    toArray() 转换为数组,hidden()不输出的字段 public function index(){ $user = model('User'); $data = $user::)-> ...

  5. 窗体Showmedol 遇到的奇怪异常: cannot make a visible window model

    //窗体Showmedol 遇到的奇怪异常: cannot make a visible window model //背景:ShowModal A窗体,A窗体再ShowModal B窗体:A是透明背 ...

  6. display:none与visible:hidden的区别

    display:none和visible:hidden都能把网页上某个元素隐藏起来,但两者有区别: display:none ---不为被隐藏的对象保留其物理空间,即该对象在页面上彻底消失,通俗来说就 ...

  7. 关于Delphi错误:Cannot make a visible window modal

    Delphi的fsMDIChild类型的窗体是不能使用ShowModal的,否则会弹出"Cannot make a visible window modal"异常, 但是把fsMD ...

  8. Android笔记——Android中visibility属性VISIBLE、INVISIBLE、GONE的区别

    在Android开发中,大部分控件都有visibility这个属性,其属性有3个分别为"visible "."invisible"."gone&quo ...

  9. Textbox.Visible=False隐藏方式导致的问题

    今天公司的正式环境有个功能不好使,但是测试环境没有问题,经过和同事的研讨,发现应该是我在写代码的时候把Textbox的visible属性设置为false导致的. 当时的需求是需要在发邮件的时候加上“相 ...

随机推荐

  1. 没有dig命令的结觉方法

    -bash: dig: command not found rpm -qa bind-utils 查看没有没安装 bind-utils 包 yum install -y bind-utils 解决

  2. Linux 查看一个端口的连接数

    netstat -antp|grep -i "80" |wc -l 譬如查看80端口的连接数

  3. python爬虫实战(3)--图片下载器

    本篇目标 1.输入关键字能够根据关键字爬取百度图片 2.能够将图片保存到本地文件夹 1.URL的格式 进入百度图片搜索apple,这时显示的是瀑布流版本,我们选择传统翻页版本进行爬取.可以看到网址为: ...

  4. eclipse 中文版 变成 英文版 方法

    找到目录运行命令 “eclipse.exe -nl en”

  5. loader的简单使用过程分析

    首先,fragment或者activity必须实现callback接口 必须实现的三个方法为 public Loader<Cursor> onCreateLoader(int id, Bu ...

  6. 【总结整理】javascript基础入门学习(慕课网学习)

    https://www.imooc.com/learn/36 注意: javascript作为一种脚本语言可以放在html页面中任何位置,但是浏览器解释html时是按先后顺序的,所以前面的script ...

  7. Codeforces 1077(F1+F2) DP 单调队列

    题意:给你一个n个元素的数组,从中选取x个元素,并且要保证任意的m个位置中必须至少有一个元素被选中,问选中元素的和最大可以是多少? F1 n,m,x到200 F2 n,m,x到5000. 思路1:设d ...

  8. Inheritance with EF Code First: Part 1 – Table per Hierarchy (TPH)

    以下三篇文章是Entity Framework Code-First系列中第七回:Entity Framework Code-First(7):Inheritance Strategy 提到的三篇.这 ...

  9. 开发一个属于自己的第一个Composer/Packagist包

    Composer 给我们带来了诸多的好处: 模块化,降低代码重用成本 统一的第三方代码组织方式 更科学的版本更新 初始化项目,生成composer.json文件 初始实例项目代码目录结构: 现在要在项 ...

  10. 对Dapper的一点改造

    微软推出的ORM, EF在我开发的项目中给我的感觉一直都是慢.优点是高度封装的底层.便于开发. Dapper在多篇性能比较的网站中.都是名列前三.缺点是手写SQL,不便于开发.   如果能结合EF的优 ...