浅尝KubeSphere,安装MySQL等中间件
本文字数 0 | 阅读时长 ≈ 0 分钟
category
技术工具
tags
云原生
k8s
实用教程
type
Post
slug
kubesphere-k8s-middleware-mysql-redis-nacos-sentinel-seata
summary
kubesphere提供了一种便携的k8s集群管理方式,借助kubesphere快速在k8s中搭建Mysql/Redis/Nacos/SEATA/Sentinel
status
Published
date
Jul 13, 2022
icon
password
前言
我们团队在服务器环境中中统一使用KubeSphere作为k8s的默认可视化界面,它可以方便地管理k8s环境中的各个元素。包括每个节点的内存、硬盘、CPU用量等等。

创建各个中间件
实际生产时,以下组件都需要做HA高可用,这里仅演示一个单机测试版本。
1. MySQL
核心三要素
部署方式(StatefulSet),数据存储 (PVC),配置文件(ConfigMap)。

开始前准备
可在官方镜像文档描述查看mysql容器启动的相关配置
传统 Docker的启动方式
docker run -p 3306:3306 --name mysql-01 \
-v /mydata/mysql/log:/var/log/mysql \
-v /mydata/mysql/data:/var/lib/mysql \
-v /mydata/mysql/conf:/etc/mysql/conf.d \
-e MYSQL_ROOT_PASSWORD=root \
--restart=always \
-d mysql:5.7创建配置configmap
- 在配置项中添加配置

- 添加一个key-value配置,key就是配置文件的文件名 这里是
my.cnf

my.cnf的内容,填写的是数据库服务端的配置
[client]
default-character-set=utf8mb4
[mysql]
default-character-set=utf8mb4
[mysqld]
init_connect='SET collation_connection = utf8mb4_unicode_ci'
init_connect='SET NAMES utf8mb4'
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
skip-character-set-client-handshake
skip-name-resolve创建 挂载卷
- 创建卷

- 设置存储类型和访问模式以及卷容量

添加工作负载
- 在部署工作负载中选择有状态负载

- 镜像 配置为dockerhub中的mysql,并使用镜像默认端口设置

- 镜像环境变量,根据mysql镜像的官方要求,需要添加一个
MYSQL_ROOT_PASSWORD;创建后点击下一步

- 挂载存储卷


- 挂载配置卷

指定配置卷的挂载目录

对外 提供服务
- KS默认会创建一个内网服务,可以选择修改,或重新创建一个NodePort对外服务

- NodePort类型表示可以从集群中的任意节点IP进行访问

- 访问地址,如图冒号后的地址就是外部访问的端口号

- 访问客户端测试

2. Nacos
开始前准备
可参考nacos官方的部署手册
部署有状态服务
虽然nacos没有存储,但是为了保证nacos的访问地址稳定需要部署成有状态(StatefulSet)

设置镜像
可在官方镜像文档描述查找nacos-server镜像,及容器启动的相关配置

- 环境变量
我这里用到了上面 的数据库 , 需要再添加一个 MODE = standalone 才能支持数据库配置,否则默认是集群模式运行

- 同步 主机时区

暴露外部服务


Nacos的一些问题
负载时的演示效果不同

nacos的默认配置文件
# spring
server.servlet.contextPath=${SERVER_SERVLET_CONTEXTPATH:/nacos}
server.contextPath=/nacosserver.port=${NACOS_APPLICATION_PORT:8848}
spring.datasource.platform=${SPRING_DATASOURCE_PLATFORM:""}
nacos.cmdb.dumpTaskInterval=3600
nacos.cmdb.eventTaskInterval=10
nacos.cmdb.labelTaskInterval=300
nacos.cmdb.loadDataAtStart=false
db.num=${MYSQL_DATABASE_NUM:1}
db.url.0=jdbc:mysql://${MYSQL_SERVICE_HOST}:${MYSQL_SERVICE_PORT:3306}/${MYSQL_SERVICE_DB_NAME}?${MYSQL_SERVICE_DB_PARAM:characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true&useSSL=false}
db.url.1=jdbc:mysql://${MYSQL_SERVICE_HOST}:${MYSQL_SERVICE_PORT:3306}/${MYSQL_SERVICE_DB_NAME}?${MYSQL_SERVICE_DB_PARAM:characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true&useSSL=false}db.user=${MYSQL_SERVICE_USER}
db.password=${MYSQL_SERVICE_PASSWORD}
### The auth system to use, currently only 'nacos' is supported:
nacos.core.auth.system.type=${NACOS_AUTH_SYSTEM_TYPE:nacos}
### The token expiration in seconds:
nacos.core.auth.default.token.expire.seconds=${NACOS_AUTH_TOKEN_EXPIRE_SECONDS:18000}
### The default token:
nacos.core.auth.default.token.secret.key=${NACOS_AUTH_TOKEN:SecretKey012345678901234567890123456789012345678901234567890123456789}
### Turn on/off caching of auth information. By turning on this switch, the update of auth information would have a 15 seconds delay.
nacos.core.auth.caching.enabled=${NACOS_AUTH_CACHE_ENABLE:false}
nacos.core.auth.enable.userAgentAuthWhite=${NACOS_AUTH_USER_AGENT_AUTH_WHITE_ENABLE:false}
nacos.core.auth.server.identity.key=${NACOS_AUTH_IDENTITY_KEY:serverIdentity}
nacos.core.auth.server.identity.value=${NACOS_AUTH_IDENTITY_VALUE:security}
server.tomcat.accesslog.enabled=${TOMCAT_ACCESSLOG_ENABLED:false}
server.tomcat.accesslog.pattern=%h %l %u %t "%r" %s %b %D
# default current work dir
server.tomcat.basedir=
## spring security config
### turn off security
nacos.security.ignore.urls=${NACOS_SECURITY_IGNORE_URLS:/,/error,/**/*.css,/**/*.js,/**/*.html,/**/*.map,/**/*.svg,/**/*.png,/**/*.ico,/console-fe/public/**,/v1/auth/**,/v1/console/health/**,/actuator/**,/v1/console/server/**}
# metrics for elastic search
management.metrics.export.elastic.enabled=false
management.metrics.export.influx.enabled=false
nacos.naming.distro.taskDispatchThreadCount=10
nacos.naming.distro.taskDispatchPeriod=200
nacos.naming.distro.batchSyncKeyCount=1000
nacos.naming.distro.initDataRatio=0.9
nacos.naming.distro.syncRetryDelay=5000
nacos.naming.data.warmup=true3. Redis
4. SEATA
部署前准备
官方文档的Docker部署方式
seata 连接到Nacos,由nacos托管 seata的配置;同时客户端是通过nacos的注册中心中获取seata的地址。
创建配置文件CM
我这里需要将SEATA注册到nacos中,使用nacos进行配置,用mysql进行undo_log的存储,需要修改
application.yml文件,配置参考如下:server:
port: 7091
spring:
application:
name: seata-server
logging:
config: classpath:logback-spring.xml
file:
path: ${user.home}/logs/seata
extend:
logstash-appender:
destination: svc-logstash:4560
kafka-appender:
bootstrap-servers: svc-kafka:9092
topic: logback_to_logstash
console:
user:
username: admin
password: admin
seata:
config:
# support: nacos, consul, apollo, zk, etcd3
type: nacos
nacos:
server-addr: nacos-svc:8848
group: SEATA_GROUP
namespace:
username: nacos
password: nacos
registry:
# support: nacos, eureka, redis, zk, consul, etcd3, sofa
type: nacos
nacos:
application: seata-server
server-addr: nacos-svc:8848
group: SEATA_GROUP
namespace:
username: nacos
password: nacos
# server:
# service-port: 8091 #If not configured, the default is '${server.port} + 1000'
security:
secretKey: SeataSecretKey0c382ef121d778043159209298fd40bf3850a017
tokenValidityInMilliseconds: 1800000
ignore:
urls: /,/**/*.css,/**/*.js,/**/*.html,/**/*.map,/**/*.svg,/**/*.png,/**/*.ico,/console-fe/public/**,/api/v1/auth/login#创建有状态副本
- 查找镜像

- 开放端口
7091是面板的地址 ,8091是 服务端口地址

- 挂载配置文件,注意,我们这里只修改
application.yml配置文件,为了不覆盖容器中/seata-server/resources目录下的其它配置文件,这里要使用子路径配置。

暴露服务

此方案部署后,seata-server虽然部署成功,但是注册到Nacos上的IP是Kubernetes内部集群IP,seata客户端应用是无法访问到seata-server服务的,除非nacos和seata客户端应用也部署在Kubernetes集群内部,但显然目前无法进行这么大的变革。
所以解决问题的方法就是让seata-server的注册IP是宿主机IP.
定义SEATA的外部IP
定义SEATA的外部端口
k8s如果用NodePort方式暴露端口的话端口必须是30000-32765之间的端口号,但是在容器中默认配置的端口是8091,这就会导致连接不上,要从外部访问的折中之计就是先利用hostPort将容器的端口直接在宿主机上暴露。

这样做会占用了宿主机唯一的8091端口,要注意实例数不能大于节点机数,否则多出来的会无法部署。
通过Mysql存储实现Seata的高可用
开始前准备
创建seata的数据库,并且执行官方的初始化脚本 mysql.sql , 数据库建完效果如下,我目前用的是1.5.2版本的脚本:

在NACOS中配置Seata的Store类型为DB
创建
/config.txt,并填写数据库配置如下:service.vgroupMapping.ruoyi-system-group=default
store.mode=db
store.db.datasource=druid
store.db.dbType=mysql
store.db.driverClassName=com.mysql.jdbc.Driver
store.db.url=jdbc:mysql://mysql-dev-svc:3306/ry-seata?useUnicode=true
store.db.user=root
store.db.password=root
store.db.minConn=5
store.db.maxConn=30
store.db.globalTable=global_table
store.db.branchTable=branch_table
store.db.queryLimit=100
store.db.lockTable=lock_table
store.db.maxWait=5000创建一个nacos配置导入脚本 /conf/nacos-config.sh 内容点击展开:
#!/usr/bin/env bash
# Copyright 1999-2019 Seata.io Group.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at、
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
while getopts ":h:p:g:t:u:w:" opt
do
case $opt in
h)
host=$OPTARG
;;
p)
port=$OPTARG
;;
g)
group=$OPTARG
;;
t)
tenant=$OPTARG
;;
u)
username=$OPTARG
;;
w)
password=$OPTARG
;;
?)
echo " USAGE OPTION: $0 [-h host] [-p port] [-g group] [-t tenant] [-u username] [-w password] "
exit 1
;;
esac
done
urlencode() {
for ((i=0; i < ${#1}; i++))
do
char="${1:$i:1}"
case $char in
[a-zA-Z0-9.~_-]) printf $char ;;
*) printf '%%%02X' "'$char" ;;
esac
done
}
if [[ -z ${host} ]]; then
host=localhost
fi
if [[ -z ${port} ]]; then
port=8848
fi
if [[ -z ${group} ]]; then
group="SEATA_GROUP"
fi
if [[ -z ${tenant} ]]; then
tenant=""
fi
if [[ -z ${username} ]]; then
username=""
fi
if [[ -z ${password} ]]; then
password=""
fi
nacosAddr=$host:$port
contentType="content-type:application/json;charset=UTF-8"
echo "set nacosAddr=$nacosAddr"
echo "set group=$group"
failCount=0
tempLog=$(mktemp -u)
function addConfig() {
curl -X POST -H "${contentType}" "http://$nacosAddr/nacos/v1/cs/configs?dataId=$(urlencode $1)&group=$group&content=$(urlencode $2)&tenant=$tenant&username=$username&password=$password" >"${tempLog}" 2>/dev/null
if [[ -z $(cat "${tempLog}") ]]; then
echo " Please check the cluster status. "
exit 1
fi
if [[ $(cat "${tempLog}") =~ "true" ]]; then
echo "Set $1=$2 successfully "
else
echo "Set $1=$2 failure "
(( failCount++ ))
fi
}
count=0
for line in $(cat $(dirname "$PWD")/config.txt | sed s/[[:space:]]//g); do
(( count++ ))
key=${line%%=*}
value=${line#*=}
addConfig "${key}" "${value}"
done
echo "========================================================================="
echo " Complete initialization parameters, total-count:$count , failure-count:$failCount "
echo "========================================================================="
if [[ ${failCount} -eq 0 ]]; then
echo " Init nacos config finished, please start seata-server. "
else
echo " init nacos config fail. "
fi执行脚本,后面跟上nacos的ip和端口,例如:
./nacos-config.sh -h192.168.31.191 -p31864
配置添加成功的效果如下:当然这些配置你也可以在NACOS中逐一添加

- 导入配置后,重新启动Seata,若日志结尾出现
{dataSource-1} inited表示Seata 成功连接mysql
06:57:28.361 INFO --- [ main] io.seata.config.ConfigurationFactory : load Configuration from :Spring Configuration
[4.265s][info ][gc] GC(8) Pause Young (Allocation Failure) 61M->19M(149M) 15.132ms
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
06:57:29.934 INFO --- [ main] com.alibaba.druid.pool.DruidDataSource : {dataSource-1} inited
06:57:30.007 ERROR --- [ main] i.s.s.s.d.l.DataBaseDistributedLocker : The distribute lock table is not config, please create the target table and config it
[5.506s][info ][gc] GC(9) Pause Young (Allocation Failure) 62M->24M(149M) 20.491ms
06:57:30.168 INFO --- [ main] i.s.core.rpc.netty.NettyServerBootstrap : Server started, service listen port: 8091
06:57:30.279 INFO --- [ main] io.seata.server.ServerRunner : seata server started in 1981 millSeconds相关
- Author:tangly1024
- URL:https://tangly1024.com/article/kubesphere-k8s-middleware-mysql-redis-nacos-sentinel-seata
- Copyright:All articles in this blog, except for special statements, adopt BY-NC-SA agreement. Please indicate the source!
Relate Posts :
Tags:
云原生
k8s
实用教程
















