架构►存储►kudu

kudu - usage

安装

新版本安装参考这里。

centos

build

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash
sudo yum -y install autoconf automake curl cyrus-sasl-devel cyrus-sasl-gssapi \
  cyrus-sasl-plain flex gcc gcc-c++ gdb git java-1.8.0-openjdk-devel \
  krb5-server krb5-workstation libtool make openssl-devel patch pkgconfig \
  redhat-lsb-core rsync unzip vim-common which
sudo yum -y install centos-release-scl-rh
sudo yum -y install devtoolset-8
git clone https://github.com/apache/kudu
cd kudu
build-support/enable_devtoolset.sh thirdparty/build-if-necessary.sh
mkdir -p build/release
cd build/release
../../build-support/enable_devtoolset.sh \
  ../../thirdparty/installed/common/bin/cmake \
  -DCMAKE_BUILD_TYPE=release \
  ../..
make -j4

对于 centos 7 默认 gcc 是 4.8，用 yum install gcc72-c++ 升级到 gcc-g++ 7.2

package

1
2
3
4
# 下载 repo 配置
$ wget http://archive.cloudera.com/kudu/redhat/7/x86_64/kudu/cloudera-kudu.repo -P /etc/yum.repos.d/
# 安装
$ yum update & yum install kudu
linux►software

linux - crontab

命令格式

1
2
3
MIN HOUR DOM MON DOW CMD
# DOM: day of month
# DOW: day of week

查看

1
$ crontab -l

编辑

1
$ crontab -e

启动时运行

1
@reboot /home/ubuntu/app/consul/start.sh

时间格式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
MIN HOUR DOM MON DOW CMD
 *   *    *   *   *   ...
 
# 特殊符号
*	任意值
,	多值分隔符
-	范围分隔符
/	间隔符 (step values)

# 示例
* * * * *       每分钟
30 08 10 06 *   每年的 6 月 10 号 8 点 30 分
30 * * * *      每小时的 30 分
00 11,16 * * *  每天的 11:00 和 16:00
00 09-18 * * *  每天的 9、10 ... 18 点 00 分
00 09-18 * * 1-5 周一至周五每天的 9、10 ... 18 点 00 分
*/10 * * * *    每十分钟

设置 PATH

1
2
$ crontab -e # 开头添加如下内容
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/home/ubuntu/.local/bin
python►notes

python - generate wheel

安装

1
$ pip3 install wheel setuptools

创建 setup.py

1
2
3
4
5
6
7
from setuptools import setup

setup(
    name='abtest',
    version='1.0',
    packages=['.abtest'],
)

setup.py 示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import os.path

from setuptools import setup

REQUIREMENTS = 'requirements.txt'
requires = []
if os.path.isfile(REQUIREMENTS):
    with open(REQUIREMENTS, 'r') as f:
        requires = f.read().splitlines()

# VERSION i.e. 1.3
with open('VERSION', 'r') as f:
    v = f.read()
    setup(
        name='abtest',
        version=v.strip(),
        packages=['abtest', 'data', 'model', 'utils'], # 指定 package
        install_requires=requires # 指定依赖基本版本
    )

打包工程

1
python setup.py bdist_wheel --universal
coding►c++

c++ compiler

指定编译器

1
2
3
4
5
export PATH=/opt/scylladb/bin:$PATH
export LD_LIBRARY_PATH=/opt/scylladb/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/scylladb/lib:$LD_LIBRARY_PATH
export CC=/opt/scylladb/bin/gcc
export CXX=/opt/scylladb/bin/g++

编译选项

1
2
3
4
5
6
# warning
-Wall			开启警告（all: 应该开启的最小警告集合）
-Wextra		开启扩展警告
-Werror 	所有警告视为错误
-Wno-error=...	关闭某项警告视为错误
-Wno-<...>			关闭某项警告视为错误

附录

-Wall

1
2
3
4
5
6
7
8
9
-Waddress -Warray-bounds (only with -O2) -Wc++0x-compat -Wchar-subscripts
-Wenum-compare (in C/Objc; this is on by default in C++) -Wimplicit-int (C and
 Objective-C only) -Wimplicit-function-declaration (C and Objective-C only) 
-Wcomment -Wformat -Wmain (only for C/ObjC and unless -ffreestanding) 
-Wmissing-braces -Wnonnull -Wparentheses -Wpointer-sign -Wreorder -Wreturn-type 
-Wsequence-point -Wsign-compare (only in C++) -Wstrict-aliasing 
-Wstrict-overflow=1 -Wswitch -Wtrigraphs -Wuninitialized -Wunknown-pragmas 
-Wunused-function -Wunused-label -Wunused-value -Wunused-variable 
-Wvolatile-register-var

-Wextra

1
2
3
4
-Wclobbered -Wempty-body -Wignored-qualifiers -Wmissing-field-initializers
-Wmissing-parameter-type (C only) -Wold-style-declaration (C only) -Woverride-init
-Wsign-compare -Wtype-limits -Wuninitialized -Wunused-parameter (only with -Wunused
 or -Wall) -Wunused-but-set-parameter (only with -Wunused or -Wall)

warn 示例配置

1
-Wall -Wextra -Waggregate-return -Wcast-align -Wcast-qual -Wdisabled-optimization -Wdiv-by-zero -Wendif-labels -Wformat-extra-args -Wformat-nonliteral -Wformat-security -Wformat-y2k -Wimplicit -Wimport -Winit-self -Winline -Winvalid-pch -Wjump-misses-init -Wlogical-op -Werror=missing-braces -Wmissing-declarations -Wno-missing-format-attribute -Wmissing-include-dirs -Wmultichar -Wpacked -Wpointer-arith -Wreturn-type -Wsequence-point -Wsign-compare -Wstrict-aliasing -Wstrict-aliasing=2 -Wswitch -Wswitch-default -Werror=undef -Wno-unused -Wvariadic-macros -Wwrite-strings -Wc++-compat -Werror=declaration-after-statement -Werror=implicit-function-declaration -Wmissing-prototypes -Werror=nested-externs -Werror=old-style-definition -Werror=strict-prototypes
coding►c++

内存泄漏排查

简述

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ pid=$(pgrep ranker_service)
$ pmap -x $pid
9085:   ./bin/ranker_service --cloud_region=aws-se --gray_cloud_list=none
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000   19180    7236       0 r-x-- ranker_service
00000000018bb000     280     168      24 r---- ranker_service
0000000001901000      92      40      40 rw--- ranker_service
0000000001918000     212     120     120 rw---   [ anon ]
0000023412800000  262144  216280  216280 rw---   [ anon ]
000002342f000000  528384  524292  524292 rw---   [ anon ]
0000023453800000  262144  262144  262144 rw---   [ anon ]
0000023469c00000  262144  262144  262144 rw---   [ anon ]
0000023486000000  262144  262144  262144 rw---   [ anon ]
00000234aec00000  524288  518296  518296 rw---   [ anon ]
00000234db400000  262144  150220  150220 rw---   [ anon ]
0000023546400000 23330816 23188160 23188160 rw---   [ anon ]
0000023b50400000  303104  294920  294920 rw---   [ anon ]
0000023b69000000  475136  458768  458768 rw---   [ anon ]
...
$ gcore $pid

dump

1
2
3
4
5
6
7
8
$ gdb bin/ranker_service core.9085
(gdb) dump binary memory result.bin 0x23546400000 0x23586400000  # dump 1G 内存数据

## dump 内存块 0x23546400000 全部数据
# 计算地址
> hex(0x0000023546400000 + 23330816 * 1024) # python3
'0x23ad6400000'
(gdb) dump binary memory result.bin 0x23546400000 0x23ad6400000  # dump 区块全部数据 23330816KB

统计 vtable

1
2
$ hexdump result.bin | awk '{printf "%s%s%s%s\n%s%s%s%s\n", $5,$4,$3,$2,$9,$8,$7,$6}' | sort | uniq -c | sort -nr  > hex.t
$ hexdump core.9085 | awk '{printf "%s%s%s%s\n%s%s%s%s\n", $5,$4,$3,$2,$9,$8,$7,$6}' | sort | uniq -c | sort -nr  | head

image-20220613180006614

image-20220613180039586

参考

linux►software

nginx creat self-signed SSL certificate

创建证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 自认证证书
# 生成 nginx-selfsigned.pass.key
$ openssl genrsa -des3 -passout pass:over4chars -out nginx-selfsigned.pass.key 2048
# 生成 nginx-selfsigned.key
$ openssl rsa -passin pass:over4chars -in nginx-selfsigned.pass.key -out nginx-selfsigned.key
$ rm nginx-selfsigned.pass.key # 可以删除了
# 生成 nginx-selfsigned.csr
$ openssl req -new -key nginx-selfsigned.key -out nginx-selfsigned.csr # 一直回车
# 生成 nginx-selfsigned.crt
$ openssl x509 -req -sha256 -days 365 -in nginx-selfsigned.csr -signkey nginx-selfsigned.key -out nginx-selfsigned.crt

# 需要的文件
- nginx-selfsigned.key
- nginx-selfsigned.crt

或者

1
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt

设置 nginx 使用 ssl

把生成的整数文件放到 /certs 目录下。

1
2
3
$ sudo mkdir /certs
$ sudo mv nginx-selfsigned.key /certs
$ sudo mv nginx-selfsigned.crt /certs

生成 dhparam.pem 文件。

1
$ openssl dhparam -out /certs/dhparam.pem 4096

创建 /etc/nginx/snippets/self-signed.conf ，输入如下内容。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
ssl_certificate /certs/nginx-selfsigned.crt;
ssl_certificate_key /certs/nginx-selfsigned.key;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;

ssl_dhparam /certs/dhparam.pem;

设置 nginx site 配置。

1
2
3
4
5
6
7
8
9
10
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;

    server_name server_domain_or_IP;
    include snippets/self-signed.conf;
    ...
}

配置 http 跳转 https

按需配置。

1
2
3
4
5
6
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name server_domain_or_IP;
    return 302 https://$server_name$request_uri;
}

参考

coding►c++►tools►gperftools

gperftools

安装

1
$ yum install gperftools gperftools-devel -y

Heap Profile

1
2
3
4
5
6
7
8
# dump
$ env LD_PRELOAD="/usr/lib64/libtcmalloc_and_profiler.so" HEAPPROFILE="heap-profile" <binary>
$ env LD_PRELOAD="/usr/lib64/libtcmalloc.so" HEAPPROFILE="heap-profile" <binary>
$ env LD_PRELOAD="/usr/lib64/libtcmalloc.so" HEAPPROFILE="/tmp/profile" HEAP_PROFILE_TIME_INTERVAL=60 <binary>

# analysis
$ pprof --text <binary> /tmp/profile....heap
$ pprof --svg <binary> /tmp/profile....heap > heap.svg
架构►container►k8s

k8s usage

[toc]

节点管理

删除节点

1
$ sudo kubeadm reset cleanup-node  # master 节点也可以清除

标签

1
2
3
4
5
# 查看 node 标签
$ kubectl get nodes --show-labels

# 打标签
$ kubectl label nodes <your-node-name> disktype=ssd

设置 pod 的 node 标签选择（nodeSelector）

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  nodeSelector:
    disktype: ssd

设置 deployment node 标签选择

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
spec:
	template:
    # 按标签选择机器
    nodeSelector:
      <label-name>: <label-value>
    # 设置机器亲和性
    affinity:
      # 设置 node 亲和性
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: <label-name>
                  operator: In
                  values:
                    - <label-value>
      # 设置 pod 亲和性
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                    - <pod-name>
                    - <pod-name>
            topologyKey: kubernetes.io/hostname

Deployment 管理

1
kubectl delete --all deployments d-n=<ns>

Pod 管理

1
2
3
4
5
6
7
8
9
10
# 查看 pod  信息
$ kubectl get pods -A  # A = all-namespaces

# 删除 pod
$ kubectl delete pod <name> --namespace=<ns>
# 批量删除
kubectl delete pod --all -n=<ns>

# 查看 pod 详情; ip 等
$ kubectl describe pod <name> --namespace=<ns>

Service

1
2
3
4
5
# 列出所有 service
k get services -o wide -A --sort-by=.metadata.name 

# 获取 service 详情; nodeport 等
k describe service kubernetes-dashboard -n kube-system

Token

1
microk8s kubectl create token -n kube-system default --duration=8544h

私有仓库

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# docker login
docker login hub.company.com

# 不指定 namespace, 设置 default
kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=~/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson

# 每个 namespace 需要单独设置
kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=~/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson \
    --namespace=ace

# 修改 deployment
apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  # 添加 imagePullSecrets
  imagePullSecrets:
  - name: regcred

Secret

1
2
3
4
5
6
7
8
9
10
11
12
# 创建
kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=$HOME/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson \
    --namespace=default

# 查看
kubectl get secrets
kubectl get secretss <secret> --output=yaml

# 删除
kubectl delete secret <secret>

Namespace

1
2
3
4
5
# 列出所有的 namespace
$ kubectl get namespaces

# 创建命名空间
$ kubectl create namespace <space-name>

登录容器

1
kubectl exec --stdin --tty <pod-instance> -- /bin/bash

工具

  • k9s，基于 SHELL 的 K8S 客户端工具
  • k8s Lens，桌面端 K8S 客户端工具
python►library►matplotlib

matplotlib

参数

alpha

  • 透明度
  • float [0, 1]

bins

  • 分段区间
  • 可以是一个整数或一个序列
  • 如果是整数n，将会均匀分为n个区间，否则按指定的序列进行分段

配置

subplot

绘制子图。

原型

1
subplot(nrows, ncols, index, **kwargs)

示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import matplotlib.pyplot as plt

y = [i for i in range(100)]
z = [i * 1.1 for i in range(100)]

plt.subplot(2, 2, 1)
plt.xlabel('x label')
plt.ylabel('y label')
plt.plot(y, label='y')
plt.plot(z, label='z')
plt.legend()

#### no legend
plt.subplot(2, 2, 2)
plt.xlabel('x label')
plt.ylabel('y label')
plt.plot(y, label='y')
plt.plot(z, label='z')

plt.subplot(2, 2, 4)
plt.xlabel('x label')
plt.ylabel('y label')
plt.plot(y, label='y')
plt.plot(z, label='z')
plt.legend()

# no NO.3 subplot 
plt.show()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# 数据来自鸢尾花数据集，绘制散点图
columns = ['Id', 'SepalLengthCm', 'SepalWidthCm', 'PetalLengthCm', 'PetalWidthCm', 'Label']
for i in range(2, 5):
    plt.subplot(1, 3, i - 1)
    plt.scatter(features[:, 1], features[:, i], c=labels, s=50, cmap='viridis')
    plt.title(columns[i])
plt.show()

# 自定义 subplot size
start = 2
end = 5
fg, axs = plt.subplots(end - start, end - start, figsize=(25, 25))
for i in range(start, end):
    for j in range(start, end):
        axs[i - start, j - start].scatter(features[:, 1], features[:, i], c=labels, s=25, cmap='viridis')
        axs[i - start, j - start].set_title(columns[i] + ' - ' + columns[j])