Darcy Tang’s Blog

记录一点编程心得

主要PaaS平台支持语言清单


AWS Elastic Beanstalk
GAE
Windows Azure
CloudFoundry
Heroku
OpenShift
SAE
BAE
tsuru(基于Juju)
Java
           x
  x
         x
         x
    x
      x
  x
  x

Ruby
           x

        
         x
    x
      x
 
 

PHP
           x

         x
         x*
  
      x
  x
  x

Python
           x
  x
         x
         x* 
    x
      x
  x
  x

.NET
           x

         x
         x*
 
   



Node.js
           x

         x
         x
    x
      x



Perl





      x



Standalone



         x

 



DIY
          x




      x


           x


  • Cloud Foundry对这些语言的支持是社区提供的。 Cloud Foundry.com支持Java、Ruby和Node,而ActiveState添加了对Python的支持,Tier 3添加了对.NET的支持,AppFog添加了对PHP的支持。

免费的AWS Elastic Beanstalk其实还包含了一种IaaS(Infrastructure-as-a-Service)产品。开发人员和管理员可以直接访问应用程序后 面的AWS基础设施,这意味着他们可以修改服务器配置或访问服务端的日志文件。用户负责各种基础设施相关的任务,包括选择(及更新)服务器的操作系统和应 用程序栈。AWS Elastic Beanstalk确实也自动化了很多管理任务,包括通过一条命令重新启动所有的web服务器、通过中心位置访问所有的服务器日志文件以及监控所有节点的性能。

OpenShift也提供对IaaS基础设施的直接访问,这样就可以让用户添加所有想要的语言和框架。其后台使用的是 Red Hat Enterprise Linux 6.2 running on x64 systems.

表格部分内容来自 InfoQ ,争取能定时更新此表格

BOSH初步研究

BOSH的组件结构

BOSH总体架构

Director

Worker

Agent

Message Bus

Health Monitor

BOSH执行部署的主要流程

BOSH release的组成

BOSH stemcell的组成

Staring an Open-Source Project

在开源中国上参与翻译了一篇如何运作开源项目的文章,把自己翻译的部分整理下发出来。 译文 原文

Tag Versions in Source Control

When you decide on a new release, use a source-control tag to mark the state of the code for that release. I started doing this for CSS Lint as soon as we started using version numbers. I didn’t think much of it until the first time I forgot to tag a release and a bug was filed by someone looking for that tag. It turns out that developers like to check out particular versions of code.

Tie the tag obviously to the version number by including the version number directly in the tag’s name. With CSS Lint, our tags are in the format of “v0.9.9.” This will make it very easy for anyone looking through tags to figure out what those tags mean — including you, because you’ll be able to better keep track of what changes have been made in each release.

Change Logs

Another benefit of versioning is in producing change logs. Change logs are important for communicating version differences to both end users and contributors. The added benefit of tagging versions and source control is that you can automatically generate change logs based on those tags. CSS Lint’s build system automatically creates a change log for each release that includes not just the commit message but also the contributor. In this way, the change log becomes a record not only of code changes, but also of contributions from the community.

Availability Announcements

Whenever a new version of the project is available, announce its availability somewhere. Whether you do this on a blog or on the mailing list (or in both places), formally announcing that a new version is available is very important. The announcement should include any major changes made in the code, as well as who has contributed those changes. Contributors tend to contribute more if they get some recognition for their work, so reward the people who have taken the time to contribute to your project by acknowledging their contribution.

Managing Contributions

Once you have everything set up, the next step is to figure out how you will accept contributions. Your contribution model can be as informal or formal as you’d like, depending on your goals. For a personal project, you might not have any formal contribution process. The developer guide would lay out what is necessary in order for the code to be merged into the repository and would state that as long as a submission follows that format, then the contribution will be accepted. For a larger project, you might want to have a more formal policy.

The first thing to look into is whether you will require a contributor license agreement (CLA). CLAs are used in many large open-source projects to protect the legal rights of the project. Every person who submits code to the project would need to fill out a CLA certifying that any contributed code is original work and that the copyright for that work is being turned over to the project. CLAs also give the project owner license to use the contribution as part of the project, and it warrants that the contributor isn’t knowingly including code to which someone else has a copyright, patent or other right. jQuery, YUI and Dojo all require CLAs for code submissions. If you are considering requiring a CLA from contributors, then getting legal advice regarding it and your licensing structure would be worthwhile.

Next, you may want to establish a hierarchy of people working on the project. Open-source projects typically have three primary designations:

  • Contributor

Anyone who has had source code merged into the repository is considered a contributor. The contributor cannot access the repository directly but has submitted patches that have been accepted.

  • Committer

People who have direct access to the repository are committers. These people work on features and fix bugs regularly, and they submit code directly to the repository.

  • Reviewer

The highest level of contributor, reviewers are commanders who also have directional impact on the project. Reviewers fulfill their title by reviewing submissions from contributors and committers, approving or denying patches, promoting or demoting committers, and generally running the project.

If you’re going to have a formal hierarchy such as this, you’ll need to draft a document that describes the role of each type of contributor and how one may be promoted through the ranks. YUI has just created a formal “ Contributor Model,” along with excellent documentation on roles and responsibilities.

At the moment, CSS Lint doesn’t require a CLA and doesn’t have a formal contribution model in place, but everyone should consider it as their open-source project grows.

The Proof

It probably took us about six months from its initial release to get CSS Lint to the point of being a fully functioning open-source project. Since then, over a dozen contributors have submitted code that is now included in the project. While that number might seem small by the standard of a large open-source project, we take great pride in it. Getting one external contribution is easy; getting contributions over an extended period of time is difficult.

And we know that we’ve been doing something right because of the feedback we receive. Jonathan Klein recently took to the mailing list to ask some questions and ended up submitting a pull request that was accepted into the project. He then emailed me this feedback:

I just wanted to say that I think CSS Lint is a model open-source project — great documentation, easy to extend, clean code, fast replies on the mailing list and to pull requests, and easily customizable to fit any environment. Starting to do development on CSS Lint was as easy as reading a couple of wiki pages, and the fact that you explicitly called out the typical workflow of a change made the barrier to entry extremely low. I wish more open-source projects would follow suit and make it easy to contribute.

Getting emails like this has become commonplace for CSS Lint, and it can become the norm for your project, too, if you take the time to set up a sustainable eco-system. We all want our projects to succeed, and we want people to want to contribute to them. As Jonathan said, make the barrier to entry as low as possible and developers will find ways to help out.

源码控制中的标记版本

当你决定要发布一个新版本时,应用源码控制标记来标注此版本的代码状态。当我们开始在CSS Lint中使用版本号,我也就开始这么做了。开始我没考虑那么多,直到有一次我忘了给一个发布版添加标记,但是发现某位开发者提交的bug却是针对那个特定标记的。这说明开发者们更倾向于检出特定版本的代码。

要让标记和版本号的绑定关系更明确,就把版本号直接包含在标记名称中。在CSS Lint中,我们的标记都使用“v0.9.9”这种格式。这样可以让每个人都能够很容易地通过标记名称来识别其含义 — 包括你自己,因为你也将能够更好地跟踪每次版本发布的改动。

变更日志

版本管理还有一个好处就是能够生成变更日志。不管是对最终用户和贡献者,变更日志都是沟通版本差异时的重要依据。版本标记和源码控制有一个附加好处,你能基于这些标记自动生成变更日志。CSS Lint的构建系统能够在每次发布是自动生成一个包含提交信息及其贡献者的变更日志。这样变更日志就不仅只是一个代码变更记录,也是社区贡献值的记录。

可用宣告

每当项目发布一个新版本时,都应该在某处发布宣告。不论是在你的博客或邮件列表或是在两者上都发布,正式宣告项目新版本已经可用是非常重要的。这份宣告应该包括项目代码的主要改动及其贡献者。对贡献者工作的认同是对他们的最大鼓励,能从贡献代码中获得更多的认同感他们就更有动力做更多的贡献。所以给予那些耗费无数精力在你的项目中的开发者以最大的赞扬吧。

管理代码贡献

万事俱备,下一步就是解决如何接受代码贡献。 你的贡献模型是非常规范还是很随意,取决于你的喜好和目标。对于个人项目,可能不需要什么规范的贡献流程。开发者指南应该说明合并代码到仓库的必要条件,一个提交先要满足这些条件才会被接受。对于更大的项目,则应该要有更多规范的策略。

首先要考虑的是是否需要一个贡献者许可协议(CLA)。CLA在很多大型开源项目中使用以保护项目的合法权利。每位提交代码的开发者都需要同意CLA,以承诺任何贡献的代码都是原创的同时将代码的版权移交给项目所有。CLA也赋予项目所有者将贡献的代码作为项目一部分的授权,而且要求贡献者保证不会故意将他人具有版权、专利或其他权利的代码包含在自己的代码中进行提交。jQuery, YUI 和 Dojo 在代码提交时都要求贡献者同意CLA。如果你正在考虑使用CLA,那么寻求一些法律咨询是很值得的。

接下来,你可能想要为项目的工作人员建立一个权限层次。开源项目一般都会设置三个主要的称号:

  • 贡献者

任何对项目做过代码贡献的人都可以算作贡献者。贡献者不能直接访问代码仓库,但是提交的补丁可以被接受。

  • 提交者

提交者有权限直接访问代码仓库。他们经常对项目做特性添加和bug修正,也能够直接提交代码到代码仓库。

  • 审查者

审查者是更高一级的贡献者,是能够对项目产生直接影响的指挥官。他们的职责就是审查贡献者和提交者提交的代码,批准或者否决补丁,任命或者撤销提交者称号,总的来说就是运作这个项目。

如果你打算采用刚才所说的权限层次,那么接下来就需要起草一份文档来描述每种类型的贡献者的角色和贡献者角色如何通过排名来进行提升。YUI最近创建了一个很正式的“贡献者模型”,有很优秀的文档来描述角色和职责。

目前CSS Lint没有CLA,也没有正式的贡献者模型,但是每个人都应该在自己的开源项目成长过程中认真考虑这件事。

证明

从CSS Lint第一次发布到形成一个全功能的开源项目大概花了我们差不多6个月时间。从那时开始,超过一打的贡献者提交的代码被接受。尽管这个数字按照一个大型开源项目的标准来说有点少,但我们仍然对此感到骄傲。获得一次外部贡献很容易,在很长一段时间内都能持续获得帮助可不容易。

而且我们明白自己做的所有努力都是正确的,原因就是我们收到很多积极的反馈。乔纳森·克莱因最近到项目的邮件列表里问了几个问题,在最后他也提交了一个pull request并被项目接受了。接着他就给我发了一封反馈邮件:

我想说CSS Lint就是开源项目的典范-文档优秀,扩展方便,代码简洁,反馈及时,定制方便。基于CSS Lint做开发就像阅读wiki一样容易,而且事 实上你提出的特有更改工作流使得项目的进入门槛变得很低。我希望有更多的开源项目能照着做,让开发者为其做共享更容易。

对CSS Lint来说收到这样的邮件已经是司空见惯的事了。如果你愿意花点时间为自己的项目建立一个可持续发展的生态系统,这种事在你的项目里也一样会成为常态。每个人都希望自己的项目能成功,都希望有大量的开发者来做贡献。但是就像乔纳森说的一样:尽量降低门槛,开发者们自然会找到方法来帮忙的。

Razor的架构研究

Razor对iPXE的使用

iPXE是Razor主要依赖的技术,配合iPXE就能实现服务器节点固件和Razor的交互。 iPXE是开源的网络启动固件,提供了全部的PXE功能,而且加入更多的高级特性, 比如:支持从HTTP服务器、iSCSI SAN、FC SAN、AoE SAN、无线网络、广域网、Infiniband网络启动;通过脚本控制boot流程。

Razor主要利用HTTP服务器启动和脚本控制两个特性。

  • 部分网卡需要特殊配置以支持通过iPXE安装Razor-Microkernel

服务器启动时

搭建iPXE服务需要DHCP服务和ftp服务,具体过程就不详述了。 服务器在进行网络启动时,iPXE会根据配置提供两种boot方式。下面的代码清单是iPXE的默认配置。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
default menu.c32
prompt 0
menu title Razor Boot Menu

timeout 50
f1 help.txt
f2 version.txt

label razor-boot
  menu label Automatic Razor Node Boot
  kernel ipxe.lkrn
  append initrd=razor.ipxe

label boot-else
  menu label Bypass Razor
  localboot 1

可以看到,使用razor-boot方式会下载tftp服务器上的ipxe.lkrn和razor.ipxe脚本文件进行启动,如下图所示

服务器会根据razor.ipxe脚本文件向Razor Server发起http请求,从这里就开始进入Razor的控制流程。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!ipxe

isset ${net0/mac} && dhcp net0 ||
isset ${net1/mac} && dhcp net1 ||
isset ${net2/mac} && dhcp net2 ||
isset ${net3/mac} && dhcp net3 ||
isset ${net4/mac} && dhcp net4 ||
isset ${net5/mac} && dhcp net5 ||
isset ${net6/mac} && dhcp net6 ||
isset ${net7/mac} && dhcp net7 ||

chain http://168.1.43.39:8026/razor/api/boot?hw_id=${net0/mac}_${net1/mac}_${net2/mac}_${net3/mac}_$
{net4/mac}_${net5/mac}_${net6/mac}_${net7/mac} || goto error

:error
sleep 15
reboot

Razor最终会返回http应答,其中包含可执行iPXE脚本,告诉服务器按何种方式启动。 比如,服务器是首次启动,则会告诉服务器下载Razor-Microkernel并启动。 下面的代码清单说明Razor返回的执行脚本内容,其中指明了kernel和initrd的下载路径

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
def get_boot_script(default_mk)
        image_svc_uri = "http://#{@config.image_svc_host}:#{@config.image_svc_port}/razor/image/mk/#{default_mk.uuid}"
        rz_mk_boot_debug_level = @config.rz_mk_boot_debug_level
        # only allow values of 'quiet' or 'debug' for this parameter; if it's anything else set it
        # to an empty string
        rz_mk_boot_debug_level = '' unless ['quiet','debug'].include? rz_mk_boot_debug_level
        boot_script = ""
        boot_script << "#!ipxe\n"
        boot_script << "kernel #{image_svc_uri}/#{default_mk.kernel} maxcpus=1"
        boot_script << " #{rz_mk_boot_debug_level}" if rz_mk_boot_debug_level && !rz_mk_boot_debug_level.empty?
        boot_script << " || goto error\n"
        boot_script << "initrd #{image_svc_uri}/#{default_mk.initrd} || goto error\n"
        boot_script << "boot || goto error\n"
        boot_script << "\n\n\n"
        boot_script << ":error\necho ERROR, will reboot in #{@config.mk_checkin_interval}\nsleep #{@config.mk_checkin_interval}\nreboot\n"
        boot_script
end

下图是Razor Server的日志,可以看到hw_id为043a的节点在通过iPXE发起boot请求后,Server返回了Razor-Microkernl的地址。

对于服务器,在收到http应答后会执行脚本到指定地址下载Razor-Microkernel并启动,下图为服务器控制台输出

OS安装过程中

在iPXE发起boot的http请求时,如果Razor Server发现node已经和policy进行了绑定,则返回http应答中包含的执行脚本会指明OS镜像的路径。 例如,安装Ubuntu时,返回的iPXE脚本如下代码清单所示,此代码清单为Ubuntu ModelTemplate的一部分

1
2
3
4
5
6
7
8
9
10
#!ipxe
echo Razor <%= @label %> model boot_call
echo Installation node UUID : <%= node.uuid %>
echo Installation image UUID: <%= @image_uuid %>
echo Active Model node state: <%= @current_state %>

sleep 3
kernel <%= "#{image_svc_uri}/#{@image_uuid}/#{kernel_path} #{kernel_args(policy_uuid)}" %> || goto error
initrd <%= "#{image_svc_uri}/#{@image_uuid}/#{initrd_path}" %> || goto error
boot

如下图所示,iPXE会到指定路径下载Ubuntu安装镜像进行启动

OS安装过程详解

Razor-Microkernel发现Server方法

MK在启动时会检测Server的IP地址并主动进行checkin操作,检测方式有两种:

  1. 在/proc/cmdline 文件中查找razor.ip或razor.server值作为Server的IP地址
  2. 使用/etc/resolv.conf中的DNS服务器IP地址作为Server的IP地址

具体代码如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
def discover_rz_server_ip
      discover_by_pxe or discover_by_dns or discover_by_dhcp
end
def discover_by_pxe
      begin
        contents = File.open("/proc/cmdline", 'r') { |f| f.read }
        server_ip = contents.split.map { |x| $1 if x.match(/razor.ip=(.*)/)}.compact
        if server_ip.size == 1
          return server_ip.join
        else
          return false
        end
      rescue
        return false
      end
end

 def discover_by_dns
      begin
        contents = File.open("/proc/cmdline", 'r') { |f| f.read }
        server_name = contents.split.map { |x| $1 if x.match(/razor.server=(.*)/)}.compact
        server_name = server_name.size == 1 ? server_name.join : 'razor'

        require 'socket'
        return TCPSocket.gethostbyname(server_name)[3..-1].first || false
      rescue
        return false
      end
end
def discover_by_dhcp
      udhcp_file = "/tmp/nextServerIP.addr"
      begin
        contents = File.open(udhcp_file, 'r') { |f| f.read }
        return contents.strip
      rescue
        return false
      end
end

Razor的程序结构

Razor的主要程序结构如下图所示:

典型流程:

  • node通过REST调用http://serverIP:8026/razor/api/node/checkin?json_hash=‘{hw_id=xxxxx,last_state=xxxx}
  • Node.js服务器解析REST请求,并执行shell命令:razor -w node checkin ‘{hw_id=xxxxx,last_state=xxxx}
  • slice负责解析此shell命令,并调用相应方法返回应答

Razor中Policy的绑定过程

先解释一下概念

  • Node:能够被Razor管理到的一台服务器就是一个Node,可以是虚拟化或物理机
  • Model:Razor中将一种OS类型的安装方式抽象为一个model,比如针对Ubuntu Precise的安装就会有一个precise的model存在,其中详细定义了OS的安装配置、安装过程、安装中的状态变化、安装中Node和Razor的交互、安装后的处理等等。model是和OS类型紧密相关的。
  • Policy:Razor根据Policy将Node和Model联系在一起。Policy与Model为1对1,与Node为1对N关系。Policy的匹配规则类似防火墙的过滤规则,具有先后优先级。优先级高的Policy会先生效,并将Model和Node绑定。
  • Active_model:Active_model是一个Policy对象,可以理解为生效的Policy。Razor发现某个Node符合一条Policy规则,则生成一个Policy对象,将状态设定为active,指明绑定的Node。之后OS的安装过程,Node都是根据这个Active_model定义的流程和规则在进行操作。

current_state与返回命令的映射表如下图:

在Windows7的Git环境中使用WinMerge作为difftool

在Git使用WinMerge作为difftool,使用Git的环境不同需要不一样的配置,而且两种配置不通用。

使用CMD

添加配置

1
2
3
git config --global diff.tool winmerge
git config --global difftool.winmerge.cmd "C:/git-difftool.bat \"$LOCAL\" \"$REMOTE\""
git config --global difftool.prompt false

也可以手动在.gitconfig文件中添加如下配置

[diff]
    tool = winmerge
[difftool "winmerge"]
    cmd = C:/git-difftool.bat \"$LOCAL\" \"$REMOTE\"
[difftool]
    prompt = false

在指定位置创建git-difftool.bat

在git-difftool.bat中添加一行

"C:/Program Files/WinMerge/WinMergeU.exe" -e -ub -dl "Base" -dr "Mine" "$1" "$2"

使用Git Bash

步骤和之前的一样,只是需要注意转义符路径的写法。

添加配置

1
2
3
git config --global diff.tool winmerge
git config --global difftool.winmerge.cmd "/C/git-difftool.bat \"\$LOCAL\" \"\$REMOTE\" "
git config --global difftool.prompt false

也可以手动在.gitconfig文件中添加如下配置

[diff]
    tool = winmerge
[difftool "winmerge"]
    cmd = /C/git-difftool.bat "$LOCAL" "$REMOTE"
[difftool]
    prompt = false

在指定位置创建git-difftool.bat

在git-difftool.bat中添加一行

"/C/Program Files/WinMerge/WinMergeU.exe" -e -ub -dl "Base" -dr "Mine" "$1" "$2"

Juju-Gui 安装笔记

在Ubuntu12.04 AMD64上安装成功,基本按照项目中的README照做就行了。

juju-gui使用了Node.jssphinx,所以需要先安装Node环境,jshint是可选的。

1
2
3
4
5
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs npm
sudo npm install -g jshint
sudo apt-get install python-sphinx

Lanunchpad下载代码

1
2
3
4
5
sudo apt-get install bzr
bzr launchpad-login yourid
bzr branch lp:juju-gui
cd juju-gui
make server

这就可以访问localhost:8888看到界面了。要能操作还需要一个Juju环境,根据文档说Juju默认版本里面没有api-server这功能,最好是使用lp:~hazmat/juju/rapi-delta这个分支。

安装Juju

1
2
3
4
cd ~
sudo bzr branch lp:~hazmat/juju/rapi-delta
cd rapi-delta
python setup.py install

然后修改用户目录下的.juju/environments.yaml,在最后添加api-port: 8081,特别要注意缩进,不然启动都出错。

再到bin目录下启动Juju

1
2
cd bin
sudo ./juju bootstrap

看到 Starting api server 就说明配置成功了

这时用netstat查看会发现8081端口并没有开始监听,需要先手动部署一个服务。

1
sudo ./juju deploy mysql

Juju默认是会到官方的Charm Store查找到mysql进行部署。

8081端口也会看到是listen状态了。

连接Gui和Juju

默认连接Juju的方式是在页面使用Websocket去连接http://localhost:8081/ws,所以需要修改配置把localhost改了,不然只能在本地访问。

修改Juju-gui里的config.js和app/config.js,把里面的localhost都改成服务器ip或者能解析的域名,重启一下服务,再访问就能看到已经有一个mysql部署好了,其他的charm也随你意安装了。