• 学派吧-由商祺云独家赞助-https://www.sq9.cn。

Ansible自动部署nginx+keepalived高可用负载均衡教程及方法

unix 学派小编 10个月前 (10-11) 565次浏览 已收录 0个评论 扫描二维码

今天学派吧给大家分享一篇关于负载均衡的教程

  • 1. 部署前准备工作
  • 2. Ansible 主机与远程主机秘钥认证
  • 3. 安装配置 Ansible
  • 4. 编写 roles,实现 web 的部署
  • 5. 编写 roles 角色部署 nginx+keepalived
  • 6. 编写 roles 角色
  • 7. 其他问题

本篇文章记录通过 Ansible 自动化部署 nginx 的负载均衡高可用,前端代理使用 nginx+keepalived,后端 web server 使用 3 台 nginx 用于负载效果的体现,结构图如下:

Ansible 自动部署 nginx+keepalived 高可用负载均衡教程及方法

1. 部署前准备工作

主机规划

  • Ansible : 192.168.214.144
  • Keepalived-node-1 : 192.168.214.148
  • Keepalived-node-2 : 192.168.214.143
  • web1 : 192.168.214.133
  • web2 : 192.168.214.135
  • web3 : 192.168.214.139

2. Ansible 主机与远程主机秘钥认证

#!/bin/bash

keypath=/root/.ssh
[ -d ${keypath} ] || mkdir -p ${keypath}
rpm -q expect &> /dev/null || yum install expect -y
ssh-keygen -t rsa -f /root/.ssh/id_rsa  -P ""
password=centos
while read ip;do
expect <<EOF
set timeout 5
spawn ssh-copy-id $ip
expect {
"yes/no" { send "yes\n";exp_continue }
"password" { send "$password\n"  }
}
expect eof
EOF
done < /home/iplist.txt

iplist.txt

192.168.214.148
192.168.214.143
192.168.214.133
192.168.214.135
192.168.214.139
192.168.214.134

执行脚本

[root@Ansible script]# ./autokey.sh

测试验证

[root@Ansible script]# ssh 192.168.214.148 'date'
Address 192.168.214.148 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Sat Jul 14 11:35:21 CST 2018

配置 Ansible 基于主机名认证,方便单独管理远程主机

vim  /etc/hosts
#
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.214.148 node-1
192.168.214.143 node-2
192.168.214.133 web-1
192.168.214.135 web-2
192.168.214.139 web-3

3. 安装配置 Ansible

#安装 ansible
[root@Ansible ~]# yum install ansible -y

#配置 ansible 主机清单
[root@Ansible ~]# vim /etc/ansible/hosts 
[all]

192.168.214.148
192.168.214.143
192.168.214.133
192.168.214.135
192.168.214.139

[node]

192.168.214.148
192.168.214.143

[web]
192.168.214.133 
192.168.214.135
192.168.214.139

#Ansible 执行 ping 测试 
 [root@Ansible ~]# ansible all -m ping

4. 编写 roles,实现 web 的部署

先看一下 web 的目录结构

[root@Ansible ~]# tree /opt/roles/web
/opt/roles/web
.
├── tasks
│   ├── install_nginx.yml
│   ├── main.yml
│   ├── start.yml
│   ├── temps.yml
│   └── user.yml
└── templates
    ├── index.html.j2
    └── nginx.conf.j2

2 directories, 7 files

按照角色执行的顺序编写

编写 user.yml

- name: create group nginx
  group: name=nginx
- name: create user nginx
  user: name=nginx group=nginx system=yes shell=/sbin/nologin

编写 install_nginx.yml

- name: install nginx webserver
  yum: name=nginx

创建 nginx 配置文件的 template 模板

由于是测试,后端 web 服务的 nginx.conf 配置文件基本保持默认,只只更具后端主机情况设置 worker 进程数,使用 ansible 的 setup 模块中的变量获取远程主机的 cpu 的数量值

#将配置文件转换成 template 文件
[root@Ansible conf]# cp nginx.conf /opt/roles/web/templates/nginx.conf.j2
#做出修改的内容如下
worker_processes {{ansible_proccessor_vcpus}};

#在 templates 目录写一个测试页内如下
vim index.html.j2
{{ ansible_hostname }} test page.

编写 temps.yml

- name: cp nginx.conf.j2 to nginx web server rename nginx.conf
  template: src=/opt/roles/web/templates/nginx.conf.j2 dest=/etc/nginx/nginx.conf
- name: cp index test page to nginx server
  template: src=/opt/roles/web/templates/index.html.j2 dest=/usr/share/nginx/html/index.html

编写 start.yml

- name: restart nginx
  service: name=nginx state=started

编写 main.yml

- import_tasks: user.yml
- import_tasks: install_nginx.yml
- import_tasks: temps.yml
- import_tasks: start.yml

编写执行主文件 web_install.yml,执行文件不能与 web 角色放在同一目录,通常放在 roles 目录

[root@Ansible ~]# vim /opt/roles/web_install.yml 


---
- hosts: web
  remote_user: root
  roles:
    - web

安装前测试: -C 选项为测试

[root@Ansible ~]# ansible-playbook -C /opt/roles/web_install.yml 

如没有问题则执行安装

[root@Ansible ~]# ansible-playbook /opt/roles/web_install.yml 

测试访问

[root@Ansible ~]# ansible web -m shell -a 'iptables -F'
192.168.214.139 | SUCCESS | rc=0 >>


192.168.214.135 | SUCCESS | rc=0 >>


192.168.214.133 | SUCCESS | rc=0 >>


[root@Ansible ~]# curl 192.168.214.133
web-1 test page.

5. 编写 roles 角色部署 nginx+keepalived

部署高可用集群需要注意各节点包括后端主机的时间问题,保证各主机时间一致。

[root@Ansible ~]# ansible all -m shell -a 'yum install ntpdate -y'

[root@Ansible ~]# ansible all -m shell -a 'ntpdate gudaoyufu.com'

6. 编写 roles 角色

编写 user.yml

- name: create nginx group
  group: name=nginx
- name: create nginx user
  user: name=nginx group=nginx system=yes shell=/sbin/nologin

编写 install_server.yml

- name: install nginx and keepalived
  yum: name={{ item }} state=latest
  with_items:
    - nginx
    - keepalived

编写 temps.yml

- name: copy nginx proxy conf and rename
  template: src=/opt/roles/ha_proxy/templates/nginx.conf.j2  dest=/etc/nginx/nginx.conf

- name: copy master_keepalived.conf.j2 to MASTER node
  when: ansible_hostname == "node-1"
  template: src=/opt/roles/ha_proxy/templates/master_keepalived.conf.j2 dest=/etc/keepalived/keepalived.conf

- name: copy backup_keepalived.conf.j2 to BACKUP node
  when: ansible_hostname == "node-2"
  template: src=/opt/roles/ha_proxy/templates/backup_keepalived.conf.j2 dest=/etc/keepalived/keepalived.conf

配置 nginx proxy 配置文件模板

[root@Ansible ~]# cp /opt/conf/nginx.conf /opt/roles/ngx_proxy/templates/nginx.conf.j2
[root@Ansible ~]# vim /opt/roles/ngx_proxy/templates/nginx.conf.j2

user nginx;
worker_processes {{ ansible_processor_vcpus }};
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.

events {
    worker_connections  1024;
}


http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;


    include /etc/nginx/conf.d/*.conf;

    upstream web {

        server 192.168.214.133:80 max_fails=3 fail_timeout=30s;
        server 192.168.214.135:80 max_fails=3 fail_timeout=30s;
        server 192.168.214.139:80 max_fails=3 fail_timeout=30s;


    }



    server {

    listen       80 default_server;
    server_name  {{ ansible_hostname }};
    root         /usr/share/nginx/html;
    index index.html index.php;

         location / {
                proxy_pass http://web;
             }

         error_page 404 /404.html;

          }


}

配置 keepalived 配置文件模板

[root@Ansible ~]# cp /opt/conf/keepalived.conf /opt/roles/ha_proxy/templates/master_keepalived.conf.j2

[root@Ansible templates]# vim master_keepalived.conf.j2 # ! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.214.1 smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 vrrp_iptables vrrp_mcast_group4 224.17.17.17 } vrrp_script chk_nginx { script "killall -0 nginx" interval 1 weight -20 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 55 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 12345678 } virtual_ipaddress { 192.168.214.100 } track_script { chk_nginx } }

同样,在 master_keepalived.conf.j2 基础修改另存为 backup_keepalived.conf.j2,只修改角色与优先级即可。注意:master_keepalived.conf.j2 文件中的检测故障降低优先级的值要确保降低后 MASTER 优先级小于 BACKUP 的优先级
编写 start.yml

- name: start nginx proxy server
  service: name=nginx state=started

编写 main.yml

- import_tasks: user.yml
- import_tasks: install_server.yml
- import_tasks: temps.yml
- import_tasks: start.yml

编写执行主文件

[root@Ansible ~]# vim /opt/roles/ha_proxy_install.yml


---
- hosts: node
  remote_user: root
  roles:
    - ha_proxy

执行检测 roles

[root@Ansible ~]# ansible-playbook -C /opt/roles/ha_proxy_install.yml 

执行测试没问题即可执行自动部署

执行过程如下:

[root@Ansible ~]# ansible-playbook  /opt/roles/ha_proxy_install.yml 



PLAY [node] **********************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************
ok: [192.168.214.148]
ok: [192.168.214.143]

TASK [ha_proxy : create nginx group] *********************************************************************************************
changed: [192.168.214.148]
ok: [192.168.214.143]

TASK [ha_proxy : create nginx user] **********************************************************************************************
changed: [192.168.214.148]
ok: [192.168.214.143]

TASK [ha_proxy : install nginx and keepalived] ***********************************************************************************
changed: [192.168.214.143] => (item=[u'nginx', u'keepalived'])
changed: [192.168.214.148] => (item=[u'nginx', u'keepalived'])

TASK [ha_proxy : copy nginx proxy conf and rename] *******************************************************************************
changed: [192.168.214.148]
changed: [192.168.214.143]

TASK [ha_proxy : copy master_keepalived.conf.j2 to MASTER node] ******************************************************************
skipping: [192.168.214.143]
changed: [192.168.214.148]

TASK [ha_proxy : copy backup_keepalived.conf.j2 to BACKUP node] ******************************************************************
skipping: [192.168.214.148]
changed: [192.168.214.143]

TASK [ha_proxy : start nginx proxy server] ***************************************************************************************
changed: [192.168.214.143]
changed: [192.168.214.148]

PLAY RECAP ***********************************************************************************************************************
192.168.214.143            : ok=7    changed=4    unreachable=0    failed=0   
192.168.214.148            : ok=7    changed=6    unreachable=0    failed=0   

至此,自动部署 nginx+keepalived 高可用负载均衡完成了

最后看一下 roles 目录的结构

[root@Ansible ~]# tree /opt/roles/
/opt/roles/
├── ha_proxy
│   ├── tasks
│   │   ├── install_server.yml
│   │   ├── main.yml
│   │   ├── start.yml
│   │   ├── temps.yml
│   │   └── user.yml
│   └── templates
│       ├── backup_keepalived.conf.j2
│       ├── master_keepalived.conf.j2
│       └── nginx.conf.j2
├── ha_proxy_install.retry
├── ha_proxy_install.yml
├── web
│   ├── tasks
│   │   ├── install_nginx.yml
│   │   ├── main.yml
│   │   ├── start.yml
│   │   ├── temps.yml
│   │   └── user.yml
│   └── templates
│       ├── index.html.j2
│       └── nginx.conf.j2
├── web_install.retry
└── web_install.yml

6 directories, 19 files

下面测试服务:keepalived 的服务没有在 ansible 中设置自动启动,到 keepalived 节点启动即可。

测试 node 节点

[root@Ansible ~]# for i in {1..10};do curl 192.168.214.148;done
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.

将 node-1 的 MASTER 服务停掉测试故障转移,同时查看 node-2 状态变化

执行: nginx -s stop

查看 vrrp 通知,可以看到主备切换正常:

[root@node-2 ~]# tcpdump -i ens33 -nn host 224.17.17.17

listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes

16:55:20.804327 IP 192.168.214.148 > 224.17.17.17: VRRPv2, Advertisement, vrid 55, prio 100, authtype simple, intvl 1s, length 20
16:55:25.476397 IP 192.168.214.148 > 224.17.17.17: VRRPv2, Advertisement, vrid 55, prio 0, authtype simple, intvl 1s, length 20
16:55:26.128474 IP 192.168.214.143 > 224.17.17.17: VRRPv2, Advertisement, vrid 55, prio 90, authtype simple, intvl 1s, length 20
16:55:27.133349 IP 192.168.214.143 > 224.17.17.17: VRRPv2, Advertisement, vrid 55, prio 90, authtype simple, intvl 1s, length 20

再测试访问:

[root@Ansible ~]# for i in {1..10};do curl 192.168.214.148;done
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.

node-1 恢复主节点,抢回 MASTER 角色

node-1 节点执行 nginx 指令,可以看到 VIP 漂移回到 node-1 节点,测试访问

[root@Ansible ~]# for i in {1..10};do curl 192.168.214.148;done
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.

7. 其他问题

上面的自动部署方式还有可以改进的地方,比如,可以将配置 keepalived 的配置文件中的许多参数在 roles 中以统一变量的方式定义,然后在 template 模板文件中引用参数就可以了

此外还有一个需要注意的地方是:keepalived 的配置文件中使用了 killall 指令检测本地的 nginx 服务状态,如果检测结果状态为非 0 就会执行 vrrp_script 中定义的降级操作,要确保系统这个指令可以执行,有时该指令没有被安装,如果该指令没有存在,即使 MASTER 节点发生故障也不会发生变化。


学派吧 , 版权所有丨如未注明 , 均为原创丨
喜欢 (0)
[sqwwnet]
分享 (0)
学派小编
关于作者:
君子爱财,取之有道!踏实做人,诚信做事! 欲全天道,先全人道,人道不能全,天道远

您必须 登录 才能发表评论!