代码实战:从单体式应用到微服务的低风险演变

11个月前

大量干货和工具:服务测试、部署和UI调整,用ISITO来管理路由以及API设计。

本文获得blog.christianposta授权翻译发表,转载需要注明来自公众号EAWorld。


作者:Christian Posta 

译者:海松 

原题:Low-risk Monolith to Microservice Evolution Part II


继续来深入探讨!在之前的文章(第一部分)中,我们为本篇文章建立了一个上下文环境(以便于讨论)。一个基本原则是,当微服务被引入到现有架构中时,不能也不应该破坏当前的请求流程(request flows)。“单体应用(monolish)”程序依然能带来很多商业价值(因此仍将在新的时代被使用,编者注),我们只能在迭代和扩展时,尽可能地减少其负面影响,这过程中就有一个经常被忽略的事实:当我们开始探索如何从单体应用过渡到微服务时,会遇到一些我们不愿意碰到的难题,但显然我们不能视而不见。如果你还没读过这段内容,我建议你再回去看看第一部分。同时也可以参考什么时候不要做微服务[0]


关注推特上的(@christianposta)或访问http://blog.christianposta.com,以获取最新更新和讨论。


在此前的第一部分,想解决的问题有: 

  • 如何可以有效可靠地生成微服务。以及如何建立一个持续交付的系统。 

  • 如何能够对服务和单体应用等对象进行测试。 

  • 如何在新的微服务中能安全地引入任何变更,包含灰度上线、金丝雀测试等等 

  • 如何将流量路由到新的服务中去,以保证启用/终止任何新的特性或更改都不会出现问题 

  • 如何面对许多棘手的数据集成挑战


一、技术层面


以下这些技术在我们的实践过程中将具备一定的指导作用:

• 开发人员服务框架(Spring Boot [1]WildFly [2]WildFly Swarm [3]) 

• API设计(APICur.io [4]

• 数据框架(Spring Boot Teiid [5]Debezium.io [6]) 

• 集成工具(Apache Camel [7]) 

• Service Mesh(Istio Service Mesh [8]) 

• 数据库迁移工具(Liquibase [9]) 

• 灰度上线/特性标记框架(FF4J [10]) 

• 部署/CI-CD平台(Kubernetes [11]/OpenShift [12]) 

• Kubernetes开发工具(Fabric8.io [13]) 

• 测试工具(Arquillian [14]Pact [15]/Arquillian Algeron [16]Hoverfly [17]Spring-Boot Test [18]RestAssured [19]Arquillian Cube [20]


我使用的是http://developers.redhat.com上的TicketMonster教程,显示从单体应用到微服务的演变,如果感兴趣的话可以关注,你还可以在github上找到相关的代码和文档(文档还在编写中):https://github.com/ticket-monster-msa/monolith


让我们一步步地读完第一部分 [21],具体来看看每一步应该怎么实施。中间还会引入上一部分中出现的一些注意事项,并在当前背景下再讨论一遍。


二、了解单体式应用


回顾下注意事项:

  • 单体式应用(代码和数据库模型)很难变更 

  • 变更需要整体重新部署和团队间高度的协调 

  • 需要进行大量测试来做回归分析 

  • 需要一个全自动的部署方式


可以的话,尽可能为单体应用安排大量的测试,哪怕不是一直有效。随着演变的开始,无论是添加新功能还是替换现有功能,我们都需要清楚了解任何更改可能产生的影响。Michael Feathers 在他《重构遗留代码》[22]的书中,将“遗留代码(legacy code)”定义为没有被测试所覆盖的代码。像JUnit和Arquillian这样的工具就很能帮到大忙。使用Arquillian,可以任意选择远程方法调用的接口的颗粒大小(fine grain or coarse grain),然后打包应用程序,不过仍需要用适当的模拟等方式,来运行打算被测试的一部分程序。例如,在单体应用(TicketMonster)中,我们可以定义一个微部署(micro-deployment),用来将原有的数据库替换为内存数据库,并预加载一些样例数据。Arquillian适用于Spring Boot应用、Java EE等。在本例中,我们将测试一个Java EE的单体架构:

public static WebArchive deployment() {  return ShrinkWrap    .create(WebArchive.class, "test.war")    .addPackage(Resources.class.getPackage())    .addAsResource("META-INF/test-persistence.xml", "META-INF/persistence.xml")    .addAsResource("import.sql")    .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml")    // Deploy our test datasource    .addAsWebInfResource("test-ds.xml"); }

更有意思的是,嵌入在运行环境中的测试可以用来验证内部工作的所有组件。例如,在上面的一个测试中,我们可以将BookingService注入到测试中,并直接运行:

@RunWith(Arquillian.class) public class BookingServiceTest {    @Deployment    public static WebArchive deployment() {        return RESTDeployment.deployment();    }    @Inject    private BookingService bookingService;    @Inject    private ShowService showService;    @Test    @InSequence(1)    public void testCreateBookings() {        BookingRequest br = createBookingRequest(1l, 0, new int[]{4, 1}, new int[]{1,1}, new int[]{3,1});        bookingService.createBooking(br);        BookingRequest br2 = createBookingRequest(2l, 1, new int[]{6,1}, new int[]{8,2}, new int[]{10,2});        bookingService.createBooking(br2);        BookingRequest br3 = createBookingRequest(3l, 0, new int[]{4,1}, new int[]{2,1});        bookingService.createBooking(br3);    }

完整的示例请参阅TicketMonster单体应用模块[23]中的BookingServiceTest。


测试的问题解决了,那么部署呢?


Kubernetes已成为容器化服务或应用程序的实际部署平台。Kubernetes处理诸如健康度检查、扩展、重启、负载平衡等事项。对于Java开发人员来说,像fabric8-maven-plugin[24]这样的工具甚至都可以用来自动构建容器或docker镜像,并生成任意部署资源文件。OpenShift[25]是Red Hat的Kubernetes的产品化版本,其中增加了开发人员的功能,包括CI/CD pipelines等。



无论是微服务、单体应用还是其他平台(比如能够处理持续的工作负载,即数据库等),Kubernetes/OpenShift都是一个适用于应用程序/服务的部署平台。通过Arquillian,容器和OpenShift pipelines,可以持续地将变更引入生产环境。顺便来看一下openshift.io[26],它将开发经验与自动CI/CD pipelines、SCM集成、Eclipse Che[27]开发人员工作区、库扫描等结合在一起。


目前,生产负载指向单体应用。如果我们翻到它的主页,我们会看到这样的内容:



接下来,让我们开始做一些改变…

三、提取用户界面UI


回顾下注意事项:

  • 一开始,先不要变更单体式应用;只需将UI复制粘贴到单独的组件即可 

  • 在UI和单体式应用间需要有一个合适的远程API—但并非所有情况下都需要 

  • 增加一个安全层 

  • 需要用某种方法以受控的方式将流量路由或分离到新的UI或单体式应用,以支持灰度上线(dark launch)/金丝雀测试(canary)/滚动发布(rolling release[28]


如果我们看下TicketMonster UI v1 [29]代码,就会发现它非常简单。静态HTML/JS/CSS组件已经被移到它自己的Web服务器,还被打包到一个容器中。通过这种方式,我们可以在单体应用之外对它进行单独部署,并独立更改或更新版本。这个UI项目仍然需要与单体应用对话来执行它的功能,所以应该是公开一个REST接口,让UI可以与之交互。对于一些单体应用来说,这说起来容易做起来难。如果你想从遗留代码中打包出来一个不错的REST API,又遇到了挑战,我强烈推荐你看看Apache Camel,尤其是它的REST DSL。


比较有意思的是,实际上单体应用并没有被改变。它的代码没有变动,同时新UI也部署完成。如果查看Kubernetes,我们会看到两个单独的部署对象和两个单独的pod:一个用于单体架构,另一个用于UI。



即使tm-ui-v1用户界面部署完了,也没有任何流量进入这个新的TicketMonster UI组件。为了简单起见,即使这个部署并没有承载生产流量,而是ticket-monster这个单体应用在承担所有流量,我们仍然可以把它当作一个简单的灰度上线。相关的UI端口仍旧可以访问:


接下来,用kubectl cli 工具从本地端口转发到特定的pod(端口80上的tm-ui-v1-3105082891-gh31x),并将其映射到本地端口8080。现在,如果导航到http://localhost:8080,应该得到一个新版本UI(注意突出显示的文本部分,表明这是一个不同的UI,但它直接指向单体应用)



如果我们这个新版本还算满意,就可以开始将流量引入进来。为此,我们将使用Istio service mesh [30]。Istio是用于管理由入口点和服务代理组成的网格控制层(control plane)。我已经写了一些关于像Envoy这样的数据层[31]以及service mesh[32]的文章。我个人强烈建议看看Istio的全部功能。接下来的几段内容,我们会围绕整个项目的全过程来依次展开讨论Istio的各项功能。如果控制层和数据层之间的区分让你困惑,请查看Matt Klein[33]撰写的博客。


我们将从使用Istio Ingress Controller[34]开始。该组件允许使用Kubernetes Ingress规范来控制流量进入Kubernetes集群。一旦安装了Istio,我们可以这样创建一个入口资源,将流量指向Ticket Monster UI的Kubernetes服务,tm-ui:


apiVersion: extensions/v1beta1 kind: Ingress metadata:  name: tm-gateway  annotations:    kubernetes.io/ingress.class: "istio" spec:  backend:    serviceName: tm-ui    servicePort: 80

一旦有了入口,就可以开始应用Istio路由规则[35]。例如,有一个规则,“任何时候有人试图与在Kubernetes中运行的tm-ui服务对话,将它们指向服务的第一版本v1”:


apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata:  name: tm-ui-default spec:  destination:    name: tm-ui  precedence: 1  route:  - labels:      version: v1

如此,我们能够更好地控制进入集群甚至深入集群内部的流量。在这个步骤的最后,我们会将所有的流量都转到tm-ui-v1部署。


四、从单体架构移除UI


回顾下注意事项

  • 从单体式应用中移除UI组件 

  • 需要对单体式应用进行最小的变更(弃用/删除/禁用UI) 

  • 不停机的前提下,再次使用受控的路由/整流方法来引入这种变更


这一步相当直接,通过删除静态UI组件来更新单体应用(删除的部分已经转移到了tm-ui-v1部署)。既然应用程序已经被释放成为一个单体应用的服务,以供UI,API或者其他一些程序调用,那么也可以对这个部署进行一些API层级的更改。而如果想对API进行一些更改,就需要部署一个新版本的UI。此处我们部署了backend-v1服务以及一个新的UI tm-ui-v2,可以利用后端服务中的这个新API。


来看看在Kubernetes集群中的部署情况:


此时,ticket-monster和tm-ui-v1正接收实时流量。backend-v1和指向它的UI--tm-ui-v2则没有流量负载。需要注意的一点是,backend-v1部署与ticket-monster部署共享数据库,但各自有略微不同的外向API(outward facing API)。


现在,新的backend-v1和tm-ui-v2组件已经部署到生产环境中。现在是时候把注意力放在一个简单而又重要的事实上:生产环境部署发生了改变,但是它们还没有发布。在turblabs.io [36]一些优秀的博客更详细地阐述了这一点[37]。现在,我们有机会部署一个非正式的灰度发布。也许我们希望这个部署慢慢来,首先面向内部用户,或者先对某个特定区域内,特定设备的部分用户进行部署等等。



既然已经有了Istio,接下来看看它能做些什么。我们只想为内部用户做一个灰度发布。我们可以用各种方式来识别内部用户,诸如headers、IP等等,在本例中,如果HTTP header带有 x-dark-launch: v2 这样的文本内容,则该请求将会被路由到新的backend-v1和tm -ui-v2服务中。以下是istio路由规则的样子:


apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata:  name: tm-ui-v2-dark-launch spec:  destination:    name: tm-ui  precedence: 10  match:    request:      headers:        x-dark-launch:          exact: "v2"  route:  - labels:      version: v2

任意用户身份登录主页时,应该可以看到当前的部署(即指向ticket-monster单体应用的tm-ui-v1):

现在,如果改变浏览器中的消息头(例如使用Firefox的修改消息头工具或其他类似工具),我们应该被路由到已灰度上线的服务(指向backend-v1的tm-ui-v2):



然后点击“开始”开始修改消息头并刷新页面:



现在,我们已经被重定向到服务的灰度发布版本。由此,可以通过做一个金丝雀发布(这里也许引1%的实时流量到新部署),来向客户群发布,同时,如果没有负面效果的话,那么就缓慢增加流量负载(5%、10%、50%等)。以下是Istio路由规则的一个例子,其将v2流量以1%进行金丝雀发布:


apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata:  name: tm-ui-v2-1pct-canary spec:  destination:    name: tm-ui  precedence: 20  route:  - labels:      version: v1    weight: 99  - labels:      version: v2    weight: 1 

能“看到”或“观察”这个版本的影响是至关重要的,稍后我们会进一步讨论。另外请注意,这种金丝雀发布方式目前正在架构外围完成,但是也可以通过istio控制内部服务间通讯/交互时采用金丝雀的方式。在接下来的几个步骤中,我们将开始看到。


五、引入新服务


回顾下注意事项

  • 我们要关注被抽取的服务的API设计或边界 

  • 可能需要重写单体式应用中的某些内容 

  • 在确定API后,将为该服务实施一个简单的脚手架或者place holder 

  • 新的Orders服务将拥有自己的数据库 

  • 新Orders服务目前不会承担任何流量


在这一步中,我们开始设计我们所设想的新订单服务的API,在做一些领域驱动设计练习时,我们常常需要确定一些边界(boundaries),新的API应该更多的与这种边界相一致。这里可以使用API建模工具来设计API,部署一个虚拟化的实施,并且随服务消费者的需求变化 一起迭代,而不是一开始花费大量的精力去构建,最后又发现需要不断修改。


在TicketMonster重构时,需要在单体应用中保留一个上文所说的API,以便在最初的服务拆分时尽可能轻松并且降低风险。无论是哪种情况,有两个给力的工具可以帮到我们:一个是网页式的API设计器,apicur.io[38],一个是测试/ API虚拟化工具,Hoverfly[39]。Hoverlfy是模拟API或捕获现有API流量的好工具,可以用来模拟mock端点。


如果我们正在构建一个新的API,或在使用领域驱动设计方法后,想看看API什么样,可以使用apicur.io工具建立一个Swagger/Open API的规范。



在TicketMonster这个例子中,我们通过在代理模式下启动hoverfly,并使用hoverfly捕获从应用程序到后端服务的流量。我们可以在浏览器设置中设置HTTP代理,从而通过hoverfly发送所有流量。这将把每个请求/响应对(request/response pair)的仿真存储在JSON文件中。这样我们就可以在Mock里使用这些请求/响应对,或者更进一步,用它们开始编写测试,以规范具体的实现代码中的一些行为。


对于所关注的请求或响应对(response pairs),我们可以生成一个JSON架构并用于测试中,参见https://jsonschema.net/#/editor


例如,结合使用Rest Assured和Hoverfly,可以调用hoverfly模拟,并确定该响应符合我们预期的JSON架构:


@Test public void testRestEventsSimulation(){    get("/rest/events").then().assertThat().body(matchesJsonSchemaInClasspath("json-schema/rest-events.json")); } 

在新的订单服务中,可以查看HoverflyTest.java [40]测试。有关测试Java微服务的更多信息,请查阅Manning这本给力的书,《测试Java微服务》[41],我的一些同事Alex Soto Bueno[42]Jason Porter[43]Andy Gumbrecht[44]也参与了这本书的撰写。


由于这篇博文已经很长了,我决定将最后的部分单独写成本主题的第三部分,其中将涉及在单体应用和微服务之间管理数据、服务消费的契约测试(consumer contract testing), 功能发布控制( feature flagging),甚至更复杂的istio路由等内容。本系列的第四部分将展示一个包含上述内容的实操Demo,使用负载仿真测试(load simulation tests)和故障注入(fault injections)。欢迎访问我的网站 [45]和关注我的Twitter [46]。


原文链接:http://blog.christianposta.com/microservices/low-risk-monolith-to-microservice-evolution-part-ii/


参考地址:

[0] http://blog.christianposta.com/microservices/when-not-to-do-microservices/

[1] https://projects.spring.io/spring-boot/

[2] http://wildfly.org/

[3] http://wildfly-swarm.io/

[4] http://www.apicur.io/

[5] https://github.com/teiid/teiid-spring-boot

[6] http://debezium.io/

[7] http://camel.apache.org/

[8] https://istio.io/

[9] http://www.liquibase.org/

[10] https://ff4j.org/

[11] https://kubernetes.io/

[12] https://www.openshift.org/

[13] https://fabric8.io/

[14] http://arquillian.org/

[15] https://github.com/pact-foundation/pact-specification

[16] http://arquillian.org/arquillian-algeron/

[17] https://hoverfly.io/

[18] https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-testing.html

[19] http://rest-assured.io/

[20] http://arquillian.org/arquillian-cube/

[21] http://blog.christianposta.com/microservices/low-risk-monolith-to-microservice-evolution/

[22] https://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052

[23] https://github.com/ticket-monster-msa/monolith/blob/master/monolith/src/test/java/org/jboss/examples/ticketmonster/test/rest/BookingServiceTest.java

[24] https://maven.fabric8.io/

[25] https://www.openshift.com/

[26] https://openshift.io/

[27] https://www.eclipse.org/che/

[28] http://blog.christianposta.com/deploy/blue-green-deployments-a-b-testing-and-canary-releases/

[29] https://github.com/ticket-monster-msa/monolith/tree/master/tm-ui-v1

[30] https://istio.io/

[31] http://blog.christianposta.com/microservices/00-microservices-patterns-with-envoy-proxy-series/

[32] http://blog.christianposta.com/microservices/application-network-functions-with-esbs-api-management-and-now-service-mesh/

[33] https://medium.com/@mattklein123/service-mesh-data-plane-vs-control-plane-2774e720f7fc

[34] https://istio.io/docs/tasks/traffic-management/ingress.html

[35] https://istio.io/docs/reference/config/traffic-rules/routing-rules.html

[36] https://www.turbinelabs.io/

[37] https://blog.turbinelabs.io/deploy-not-equal-release-part-one-4724bc1e726b

[38] http://www.apicur.io/

[39] https://hoverfly.io/

[40] https://github.com/ticket-monster-msa/monolith/blob/master/orders-service/src/test/java/org/ticketmonster/orders/HoverflyTest.java

[41] https://www.manning.com/books/testing-java-microservices

[42] https://twitter.com/alexsotob

[43] https://twitter.com/lightguardjp

[44] https://twitter.com/andygeede?lang=en

[45] http://blog.christianposta.com/

[46] https://twitter.com/christianposta


关于EAWorld微服务,DevOps,数据治理,移动架构原创技术分享,长按二维码关注

COMMENTS

2 个回应


Normal

ALEXISHor 2018年11月17日 01:02

laws
Que hacemos del 17 al 19 de septiembre LasVegasCarLoanscarregistrations ёлки 5 смотреть онлайн ёлки 5 смотреть онлайн kino ёлки car lease deals car leasing with part exchange lease not buy car 3000 BEAUFORT WRESTLING UPDATE 12/18/11 11 tips on privately selling your used car abc news vehicle values Lego Jurassic World
SixtRentaCar-13PhotosRental-Westchester-LosAngeles,CA-Reviewscartrailer JYD would be envious auto dealer websites car sales website creator dealerplatform auto dealer websites car dealership websites car sales websites car sales websites 2 o2 sensor part number for 3 0r personal car leasing personal lease deals contract hire leasing options uk car leasing uk car leasing uk New Announcements Board CheapCarRentalMelbourne,CheapCarHireMelbourneAirport,CarRentallocalcarsforsale Tyre Size quandaries ford shelby gt interviews book review editorials Simulating the dynamic boundaries of a 3D soil Model jersey city nj mold removal and mold remediation mold removal mold remediation black mold mold cleanup how to get rid of mold jersey city nj bayonne nj jersey city nj Miami Valley 4 Wheelers Spring Swap Meet Middletown Ohio Bailbondspomonabailbondspomona lease mobile porque se ha cerrado el post J5 UD Las Palmas Granada CF TotalQualityManagement(TQM)conceptsoftotalqualitymanagement 01 Nissan Pathfinder LE 2WD/auto/leather car transport used cars raleigh nc windows 7 very slow blue book value of truck used cars in CAC Hruљica 27 5 2018 CarCover,TruckCovers,CustomCoverssellcar QUESTIONS about using the forum automobile insurance information guide pay as you go car insurance pay as you go car insurance RECHERCHE CAPOT AS1000 car insurance rates mobile form compare com car insurance compare com car insurance Wei Jang's ARK SmartLease,AffordableShortTermCarLeaseinSiliconValley,shorttermcarlease.Shorttermcarlease Wo finde ich meinen Blog park dental associates inc dent general dentistry 480 park st stoughton ma phone number yelprecommendationsan francisco bay area localbusinessreviewfriendrestaurantdentistdoctor Novorocna sutaz s firmou www autoatak sk what s next for early childhood education in america online schooling for early childhood education Love Thyne Neighbor grievances disapproving quirks open V PennStateOnlinehumanresourcesandemploymentrelations money mortgage Rough Idle Yet Again ExtendedWarrantyPlanscar.comreviews X Feathers a new project a guide to common tree diseases mold removal san diego Wszystko dziala chyba oprocz Power Managmentu auto accident laws auto accident laws auto accident laws how can i create shortcuts CompareCarSizessmallcars FileSonic com полу официальная тема ex SharingMatrix com car mats bespoke personalised car mats autostyle individually tailored car mats car city Artega auf dem Genfer Automobilsalon buy used pre owned luxury cars in delhi india big boy toyz used automobiles used automobiles World of Warcraft Accounts Hacked Auction House Taken Offline FindUsedCarsforSalebyOwnerinFiveSimpleStepscarhirealicante "Its what they call ""feelings"" " backdoor iras another way to save on taxes in retirement portfolio perspective special reports portfolio perspective commentary 401k roth ira business news Modifying hovercycles and recommendations lojack lojack stolen vehicle recovery system for cars trucks classics motorcycles laptops car jack car jack ASUS ENGTX260MATRIX Video Card Review PublicCarAuction-EverySaturday10:30AMInterstateAutoAuction,carsonline.Carsonline pet property Bloddej med kerner og fro SantaMaria-AboutUsmarianhospitalsantamariaca EDC by xKrzychu how to compare car insurance rates effectively allstate hertz car rentals MTC MIDI Time Code / ReWire steam wash machine steam car washer cleaner car cleaning detailing equipment seven carwash korea car wash equipment car wash equipment Tonight I apply to grad school CarPricingArchives.eurocarhire KPT Kampfers Pocket Tool online college degrees diplomas american business and technology university university of business and technology Череповец и область Вологодская online surveys online survey tool online survey software web survey tool best survey tool best survey tool Server Side Includes OrlandoCarRental-CheapDeals-SixtRentACarusedcarsonline The Right Hand Guy gps car dvd player car dvd navigation auto dvd gps car dvd player car dvd player HWBOT Country Cup 2009 auto body kits car parts accessories spoilers and police headlights car accessories car accessories Show Active Topics ERROR WellsFargo-Personal&BusinessBanking-Student,Auto&HomeLoans-Investing&Insuranceusedsuv questions quote 911uk Classifieds the advert number 993 is a HowtoLeaseaUsedCar-bycarbra Upgrading head unit from factory widescreen va home loan rates va streamline refinance va cash out refinance va home equity loan JUDGE NENCINI SENTENCING REPORT used cars for sale by owner sell my car free bill of sale car bill of sale car internetisuhtlus asjatu lootus UsedCarsforSale-UsedCarClassifiedsbyState,usedcarsbyowner.Usedcarsbyowner 12/19/07 Gears of War best new car lease deals august 2015 car sponsors Walter and Seattle kachar s used cars used cars monroe mi dealer how to rent a car TFL store update 101609 Georgia-sIndependentRentalCompany-ECONOMYRENT-A-CAR,economycarrental.Economycarrental Bezmaksas Forex seminari iesacejiem door magnets are long lasting full magnet signs tucson budget car insurance August 8 2010 second hand car prices car rentals com Ogniwa 18650 z blaszkami i bez CalculatetheTrueDealerCostofanyNewCarfreecargames reply retail question about shipping weight BMWMercedesAudiPorscheUsedCarsDealerWashingtonDCBaltimoreMDautofinder The Official KWC 2013 Awards fourth street auto used cars louisville ky dealer used cars seattle Gothic Halloween 2013 Die Fotos von “Munster morbid” housing grants for disabled veterans home loans specially adapted housing sah sha adapted housing adapted housing grants remodeling existing home adapt an existing home sa Attack of the Time Lord Atari 2600 Version suggestion CarPaymentEstimator-LeaseandFinanceOptionscheapestnewcar Iphone 6 запакован с отпечатком Apple iosстекло autofusion car dealer websites custom auto dealer websites car parking games Kardanet mu poisil on keegi teine low mileage used classic porsche for sale sloan cars country cars country cars 06 Apr 2018 23 37 ClassicCarInsuranceconsumerreviewscars Item missing from client chevrolet cars trucks suvs crossovers and vans cars cars 2 VENTE ENCEINTES PACIFIC 3 biotech industry biotech news biotechnology articles quality management articles quality management articles "patatas a la hipertonta version facil de ""a la importante""" UsedCarWarrantyPrices:AllAboutCars:Reviews,Comparisons,Tests,andModeleurocarhire savings stock DBVertGridEh вычисляемые поля не вычисляются AudioExpress:CarAudioLowestInstalledPriceGuaranteedimportcarsfromjapan IPB email verification system cheap car hire best rental prices car hire cheap car hire car rental uk rent a car car rentals uk car car cheap car rentals spain cheap car rental usa carrentals re Модерирование 02 04 2007 cbc news hydrogen fuelled cars back on the horizon as technology improves costs drop hydrogen cars hydrogen cars 2 The Post Orgasm Check In Thread SongsofLoveFoundationgovernmentcarauctions VREMEPLOV Snijeg u Hrvatskoj money instructor personal finance business careers life skills lessons car finance car finance Using quotes lists coloured and different sized text uri masters programs uri masters programs Neues Marilyn Manson Album 2015DetroitAutoShow,Day1:EverythingYouNeedtoSee–News–CarandDrivercheckersautoparts Izotope and OS El Capitan stand by… cheap flood insurance cheap insurance arizona Keeping Sht Fresh Packages how to price a car for sale budget car insurance 14 settembre 2008 XXVIII Discesa del Ticino da Vigevano a Pavia BankReposInAlabamacarvaluations tablet tickets fizik arione or prologo saddle for road bike PersonalCarLeasingandContractHirehybridcar Base Scheme Request trade in your car europa car hire best turbo rear gear windows 7 how to configure uac user account control uac virtualization interface Serato Rane SL1 Run on FTP SepticSystemCost–AvoidReplacingYourSepticSystemsepticsystemcost,septicsystem No buying/bidding parts via pm used oem auto parts salvage yards asheville western nc local used auto parts sales asheville nc speedys western oem engines transmissions body parts tires wheels car buyer Mac OS X Tiger 10 4 nie obsluguje plikow wiekszych niz 4GB what does an assistant attorney do assistant attorney general arizona what does an assistant attorney do HOW TO KEEP LCD IN 2005 2009 MAZDA 3 ALL MODELS FloridaSunDiscountCarRentalusedcarslasvegas King original 9 reunite to car transport car delivery car collection fleet delivery dealer transfer uk car rental comparison North Korea tests new type of rocket engine car trailers for sale by trailer sales of michigan car trailers for sale car trailers for sale Fiat Stilo RICAMBI DAYLINE SCARICO CERCHI 17 GoodUsedCars,Inc.lookingforusedcars trading travel Saw a vole toay CarHireMenorcawithNohiddenCostsfrom64EURweekcarwrapping Problem z instalacja Mac OS Samsung Rv510 international centre for trade and sustainable development trade in value of car trade in value of car DLP projector suggestions eaves definition of eaves by the free dictionary roof eave Pasarela en Valencia estoy en la Marina Real FireRestorationAtlantafirerestorationatlanta,atlantafire tydrukutega suhtlemine 5 klassis porsche tails wings and spoilers body kits for 911 930 993 996 e36 e46 bmw body parts lexan race glass windshield car spoilers car spoilers savage axis 223 used cars japan japanese used cars auto craft japan used cars usa used cars usa TFL Store update 040213 HowtoFind0%FinanceDealscartires 600 years of human evolution in 3 weeks best online universities in maryland online universities in maryland atari mailing box netflix taps ritesh batra to direct robert redford and jane fonda film batra vision Piastre per bauletti per modelli Burgman dal 1998 al 2010 ThebestwintercarkitsonsaleintheUK-ChronicleLivecarsbydealers vds vps Free Movie and Learn About NEW Mortgage Rules GraduateProgram:Welcome:DepartmentofMathematics-NorthwesternUniversitynorthwesterngradschool Cross platform gui 100 free lead tracking software lead generator software YARM stuff into BW lojack lojack stolen vehicle recovery system for cars trucks classics motorcycles laptops car jack car jack Appel des medecins belges 10TipsforaSuccessfulCar-BuyingExperienceonCraigslist–Feature–CarandDriverfiatcars 2002 Blade Stryker X thrifty car rentals rental car deals car supermarkets uk Corolla ke30 adaptacion embrague hidraulico food allergy to legumes cross reactivity to pollen thomas leo ogren opals tree pollen healthy landscaping allergy books allergy prevention asthma prevention pollen pollution How To Hide Left Sitebar OlxKeralaUsedCarsbestusedcarsunder10000 AutoFlight AR Drone 2 0 control program for PC auction classic cars auction cars auction cars Оплакване от админ orthodontist virginia beach drs college in virginia beach college in virginia beach Z Out WHDload/JST with no music probs FreeCarInsuranceQuotesOnline-OnlineAutoInsurancelocalcarsforsale india finance nigeria finance HEX USBCAN VCDS Cable Genuine RossTech UsedCarsforSale,FREECarAds,Buy-SellUsedCars,UsedCarSearch. Any reason vb org hasnt upgraded guide to buying a used car without going crazy used car worth Triaxial calibration with PDMY02 material economy car hire in australia low cost car rental europcar economy car rentals economy car rentals Saturday Oct 1 Johnson Creek Ride UsedCarsForSaleByPrivateOwnerUnder1000:AllAboutCars:UsedCars,Reviews,Comparisons,Tests,andModelcarsalesadelaide Problem to convert MyBB 1 6 3 to phpBB 3 1 2 motorcars of south texas used cars san antonio tx dealer used car values Carlos Montes Casino master programme in international tax law and eu tax law 2017 masters in tax law Inline 6 headers CarSpoilerscarsaleswebsites Tire Tech Thread domed stickers home page domed stickers stickers for cars stickers for cars RBF Please Give Information traders auto sales auto traders auto traders wtf why do all My forsake adds keep disappearing UsedEngines-LowMileageUsedCarandTruckEnginesusedcarfinder usa finance england finance POST DE QUEJAS / SUGERENCIAS DE MODERACION Bestelectriccarsforkidsin2015-2016carjam GELOST Hilfe meine VIS spinnt troubleshooting hvac air flow systems troubleshooting hvac DW coffee morning Sat 24th March orlando car rental from 10 brisbane car auctions Try to look at it yourself Top10CheapUsedCarsunder1000DollarsReviews2015cardrivinggames I miss Arc top 10 used car websites in uk car insurance uk Anyone Else Feel like Fringe is pulling a DC Reboot department for transport car transport car transport 2 September 16 2009 Fivefastwaystogetreplacementcarkeys.priceofusedcars X2 не проигрывает загруженное видео cars for sale used cars for sale australia ads australia cars for sale used cars for sale classifieds australia cars for sale used cars for sale free ads australia ads for sale australia ads to 3 Help with uploading photos studebakers antique photos old cars classic vintage autos automobile clubs tours pictures antique cars antique cars Error loading Tesco Digital mp3s Havanacarhire,carrentals.Carrentals uk finance philippines finance New Kia Stinger Titanium Exhaust NewCars,SUVs,TrucksforSaleinWhitby,OntarioHyundai,0financingonnewcars.0financingonnewcars Is it possible to copy subforums sport management sport management degree online Thank you from Europa Parts used car truck suv dealership w auto sale FXR Gear 30 to 70 OFF VaginalPlasticSurgery:VaginoplastyandLabiaplastyProceduresvaginoplasty,labiaplasty,procedures,cosmeticsurgery, Odkrecenie gornej poduchy silnika renaissance rehabilitation and nursing care center renaissance care and rehabilitation center Spells that effect armys 5 tips for choosing the best mesothelioma lawyer best mesothelioma lawyer USALS showbox 300 Platinum Bookvalues-Whatismybookworth?usedcarsdenver Classic not Plastic Redcliffe January 20 2018 discount electric remote control cars from rc hobby explosion your rc car superstore online car auctions "Live Chat Thread ""The Bishop Revival""" car dvd players reverse camera kit car entertainment system elinz car dvd player car dvd player Волк / Пастель GemStatewidePlumbingEnglewood,PortCharlotte,SarasotaPlumberplumbing,plumber,clogs,cloggedtoilet,sinks,faucets,bathtubs,showers,toilets,sewerlines,watersoftener,garbagedisposal,soft germany finance canada finance It was the meter all along TopScrapCarPricesToronto,JunkCarsToronto,FastCashforCarsToronto,scrapcarprices.Scrapcarprices como se puede joder la culata con 68000 km cheaper second hand imported cars a step closer car financing Давайте раскрутим сайт Сами emergency medical services ems data systems emergency medical systems emergency medical services data systems Chaqueta Helly Hansen H/h Mod Transat 2 Nueva sin Estrenar LowCostAutoInsurancemobileautorepair Suggestion Quick Command to View F Profile jadta fine cars inc used cars scranton pa dealer renault cars 21 12 2016 17 44 cars for sale buy a new or used car online used cars online used cars online 3 Nie wbijaж gwoјdzi w akumulator InsuranceforWomen-1stforWomenInsurancehowmuchismycar работа с транзиентами в аудио dish satellite tv vs directv comparison best dish network deals dish vs directv May 18 2017 isuzu cars reviews prices motor trend isuzu cars isuzu cars 8 Minuten Ablenkung gefallig TradeMealternatives france finance pakistan finance Peter Bishop and his storyline CarMagnets&VehicleMagnetsforPoliticalCampaignsautosales "Clues Eastereggs 2 17 ""White Tulip""" something awful the internet makes you stupid somethingawful something awful frontpage news comedy games movies flash video forums WRB Bowtech Banner small businesses pbx phone system matrix telecom solutions best business pbx phone systems enterprise pbx small business pbx Dark Elf Back Catalogue 2017ChevroletCamaroZL1ConvertibleFirstDrive2017chevroletcamarozl1convertible, CAPS meeting HOL Database Entry laser technician schools beauty schools in baton rouge nagle zaczeіa ¶wieciж bardzo sіabo car insurance vancouver wa daily auto insurance 16 Apr 2017 16 38 CRIMELAB-DogPoopDNATestingServicecarinsuranceontario TFF Jim Teeny wading boots we help you sell sell your car sell your car Siteme kay t yap lam yor classic cars for sale on car and classic use car for sale use car for sale What CC is on the V5C CarInsuranceOnlineInsurancePolicyIndia-MotorInsurancebarbiecar australia finance italy finance future decks pro 25 crash and dont load itunes AppliedBehaviorAnalysisappliedbehavioranalysiscareer Heroes of Might and Magic II how to take over car payments when buying a used car local cars for sale MVS File List Parser christian a bankruptcy lawyers in philadelphia Art work for my promotional card and possibly website Home,UniversityofNewEnglandinMaine,TangierandOnline,universityofmainegraduateprograms.Universityofmainegraduateprograms Self made tools / Peliin vasatyt tyokalut binghamton lawyer personal injury lawyer car accident attorney wrongful death binghamton lawyerattorneycar accident attorneypersonal injurymalpractice MacBook Air 11 Early 2015 mobile app development platform mobile application development java Genova carbono ligero Uk HawaiiDiscountCarRentals,discountcarrentals.Discountcarrentals Infopoint Moldweb a Chisinau multiple car insurance state farm car registration check Mina El Guerra Santa Rosa de Osos Antioquia Colombia 97 locations brown mackie college locations brown mackie college 20th July 2016 Mumbai Consulate TU Darmstadt iCE CarRentalFrancemercedescars japan finance netherlands finance VAR Backtesting Notes pages 11 and 13 TheEasyWayToSellYourCar COLOR AMARILLO VUELVE automobile insurance information guide pay as you go car insurance pay as you go car insurance new guy here who needs some split port write ups using f150 parts school of art design www artschools com Simon and Garfunkel to tour CarLoansCalculatorcarsalesuk memeber up email insurance categories ukmotorists car scrappage scheme help getting started with development cars under 500 dollars used car auctions How To Unlock BlackBerry Motion BuyMyCarPerthtoycars "Official 2 14 Promotional Photos ""Jacksonville""" used cars albuquerque car insurance Beginner searching for a training plan to get fit cheapest new cars for 2014 under 14000 top 5 used trucks for sale TORINO Cerco officina autorizzata CheapCarRentals,BestPricesGuaranteed,nationalcarrental.Nationalcarrental south africa finance spain finance Eurounije co nas ceka tesi a sere JWSportCarscarleasing Photo Gallery error 5 emergency management jobs emergency management degrees online 5 emergency management jobs Cicada Fire Rescue Suggestion Form new cars used cars for sale car prices reviews at new cars for sale new cars for sale 2 Working of the Automatic Gain Control AutoElectronics.usedvehiclevalues Chat Monday Night car tinting deals in dubai uae 3m car tint window tinting services car tinting car tinting phpBB 3 1i Turkceye Cevirme used cars online battery chargers for cars more blue lights Howlongdoesacar-sairfilterlast?usedcardeals Permessi che vengono sovrascritti southside auto auctions auto auction Advertising/Selling on the forum what are the types of disability insurance iii purchase disability insurance purchase disability insurance 2015 Photo Competition 6 July/August Entries Bestconvertiblecars2017,AutoExpress,convertiblecars.Convertiblecars turkey finance poland finance iPhone SE срочно куплю AFFORDABLEAUTOINC-UsedCars-GreenBayWIDealersecondhandvehicles DIY Gadgets Bugigangas tipo Faca Voce Mesmo best suvs new suv 2015 models best rated suvs car facts Migration from Vanilla Forums v2 1 x to phpBB 3 1 x virtual dispatching solutions dispatching program question about props EducationandDegreeRequirementstoBecomeaTherapistfamilyandmarriagetherapistdegree Report Bugs Here chubb travel insurance hong kong travel insurance to japan travel insurance to japan revoluciones y consumo motor 1 4 india by car and driver day tours jaipur address phone number taxi shuttle reviews car supermarket london TOSEC ISO new maintainer ClassicCarsusedpickuptrucks Proszк o dodanie stylu dla platformy wxv pl auto window tinting cheap cars under 1000 Pantalla hora / Motor police auctions online auctions government auctions auto auctions los angeles vehicle auctions what is the value of my car what is the value of my car Fior di Latte Eis Sixwaystoincreaseyourcarsresalevaluecarsearchengines china finance sweden finance Маркетинг на продуктах питания Sportsgamesbestsmallcar Aimbot on Gamezone server legal assistant hunter college online legal assistant course pay to be on site car insurance northern ireland compare cheap ni quotes cheap car insurance ireland cheap car insurance ireland Part Out 2011 Bmw 33is CARGAMES-RACINGGAMES-BIKEGAMES-DRIVINGGAMESukcars Sam Pilgrim MOTORIZED boston web designer boston web designer web design boston ma ARDrone Flight Now on Amazon AppStore finding a pro bono divorce lawyer other free divorce resources divorce layer EDC by goat vomit Instructables-Howtomakeanything,matchboxcars.Matchboxcars 04 GT Liberty Misfire accredited online biology degree programs biology degree online accredited Series Finale 5 12 5 13 Press Release texas rn programs accredited rn schools in texas tx rn programs in texas texas rn schools I misteri dellonline CarBuyersILeuropecar cameroon finance

回复


需要 后方可回复
如果没有账号可以 一个帐号。