IT Cloud

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl get svc

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT (S) AGE

frontend LoadBalancer 10.55.242.137 35.228.73.217 80: 32701 / TCP, 8080: 32568 / TCP 4m

kubernetes ClusterIP 10.55.240.1 none> 443 / TCP 48m

Now we can create identical copies of our clusters, for example, for Production and Develop, but balancing will not work as expected. The balancer will find PODs by label, and PODs in both production and Developer clusters correspond to this label. Also, placing clusters in different projects will not be an obstacle. Although, for many tasks, this is a big plus, but not in the case of a cluster for developers and production. The namespace is used to delimit the scope. We use them discreetly, when we list PODs without specifying a scope, we are displayed by default , but the PODs are not taken out of the system scope:

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl get namespace

NAME STATUS AGE

default Active 5h

kube-public Active 5h

kube-system Active

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl get pods –namespace = kube-system

NAME READY STATUS RESTARTS AGE

event-exporter-v0.2.3-85644fcdf-tdt7h 2/2 Running 0 5h

fluentd-gcp-scaler-697b966945-bkqrm 1/1 Running 0 5h

fluentd-gcp-v3.1.0-xgtw9 2/2 Running 0 5h

heapster-v1.6.0-beta.1-5649d6ddc6-p549d 3/3 Running 0 5h

kube-dns-548976df6c-8lvp6 4/4 Running 0 5h

kube-dns-548976df6c-mcctq 4/4 Running 0 5h

kube-dns-autoscaler-67c97c87fb-zzl9w 1/1 Running 0 5h

kube-proxy-gke-bitrix-default-pool-38fa77e9-0wdx 1/1 Running 0 5h

kube-proxy-gke-bitrix-default-pool-38fa77e9-wvrf 1/1 Running 0 5h

l7-default-backend-5bc54cfb57-6qk4l 1/1 Running 0 5h

metrics-server-v0.2.1-fd596d746-g452c 2/2 Running 0 5h

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl get pods –namespace = default

NAMEREADY STATUS RESTARTS AGE

Nginxlamp-b5dcb7546-g8j5r 1/1 Running 0 4h

Let's create a scope:

esschtolts @ cloudshell: ~ / bitrix (essch) $ cat namespace.yaml

apiVersion: v1

kind: Namespace

metadata:

name: development

labels:

name: development

esschtolts @ cloudshell: ~ (essch) $ kubectl create -f namespace.yaml

namespace "development" created

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl get namespace –show-labels

NAME STATUS AGE LABELS

default Active 5h none>

development Active 16m name = development

kube-public Active 5h none>

kube-system Active 5h none>

The essence of working with scope is that for specific clusters we set the scope and we can execute commands specifying it, while they will apply only to them. At the same time, except for the keys in commands such as kubectl get pods I do not appear in the scope, therefore the configuration files of controllers (Deployment, DaemonSet and others) and services (LoadBalancer, NodePort and others) do not appear, allowing them to be seamlessly transferred between the scope, which especially relevant for the development pipeline: developer server, test server, and production server. Scopes are set in the cluster context file $ HOME / .kube / config using the kubectl config view command . So, in my cluster context entry, the scope entry does not appear (default is default ):

– context:

cluster: gke_essch_europe-north1-a_bitrix

user: gke_essch_europe-north1-a_bitrix

name: gke_essch_europe-north1-a_bitrix

You can see something like this:

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl config view -o jsonpath = '{. contexts [4]}'

{gke_essch_europe-north1-a_bitrix {gke_essch_europe-north1-a_bitrix gke_essch_europe-north1-a_bitrix []}}

Let's create a new context for this user and cluster:

esschtolts @ cloudshell: ~ (essch) $ kubectl config set-context dev \

> –namespace = development \

> –cluster = gke_essch_europe-north1-a_bitrix \

> –user = gke_essch_europe-north1-a_bitrix

Context "dev" modified.

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl config set-context dev \

> –namespace = development \

> –cluster = gke_essch_europe-north1-a_bitrix \

> –user = gke_essch_europe-north1-a_bitrix

Context "dev" modified.

As a result, the following was added:

– context:

cluster: gke_essch_europe-north1-a_bitrix

namespace: development

user: gke_essch_europe-north1-a_bitrix

name: dev

Now it remains to switch to it:

esschtolts @ cloudshell: ~ (essch) $ kubectl config use-context dev

Switched to context "dev".

esschtolts @ cloudshell: ~ (essch) $ kubectl config current-context

dev

esschtolts @ cloudshell: ~ (essch) $ kubectl get pods

No resources found.

esschtolts @ cloudshell: ~ (essch) $ kubectl get pods –namespace = default

NAMEREADY STATUS RESTARTS AGE

Nginxlamp-b5dcb7546-krkm2 1/1 Running 0 10h

You could add a namespace to the existing context:

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl config set-context $ (kubectl config current-context) –namespace = development

Context "gke_essch_europe-north1-a_bitrix" modified.

Now create a new cluster in the scope dev (it is now the default, and it can be omitted –namespace = dev ) and removed from the field by default visibility default (it is no longer the default for our cluster, and it is necessary to specify –namespace = default ):

esschtolts @ cloudshell: ~ (essch) $ cd bitrix /

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl create -f deploymnet.yaml -f loadbalancer.yaml

deployment.apps "Nginxlamp" created

service "frontend" created

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl delete -f deploymnet.yaml -f loadbalancer.yaml –namespace = default

deployment.apps "Nginxlamp" deleted

service "frontend" deleted

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl get pods

NAMEREADY STATUS RESTARTS AGE

Nginxlamp-b5dcb7546-8sl2f 1/1 Running 0 1m

Now let's look at the external IP address and open the page:

esschtolts @ cloudshell: ~ / bitrix (essch) $ curl $ (kubectl get -f loadbalancer.yaml -o json

| jq -r .status.loadBalancer.ingress [0] .ip) 2> / dev / null | grep '

'

Welcome to github.com/mattrayner/docker-lamp "target =" _blank "> Docker-Lamp aka mattrayner / lamp

Customization

Now we need to change the standard solution to our needs, namely, add configs and our application. For simplicity's sake, we'll mark (change the default) .htaccess file at the root of our application , making it simple to place our application in the / app folder . The first thing that begs to be done is to create a POD and then copy our application from the host to the container (I took Bitrix):

While this solution works, it has a number of significant disadvantages. The first thing is that we need to wait from outside by constantly polling the POD when it will raise the container and we will copy the application into it and should not do this if the container has not been raised, as well as handle the situation when it breaks our POD, external services, can rely on the status of the POD, although the POD itself will not be ready yet until the script is executed. The second drawback is that we have some kind of external script that needs to be logically not separated from the POD, but at the same time it needs to be manually launched from outside, where it is stored and somewhere there should be instructions for its use. And finally, we can have a lot of these PODs. At first glance, the logical solution is to put the code in the Dockerfile:

esschtolts @ cloudshell: ~ / bitrix (essch) $ cat Dockerfile

FROM mattrayner / lamp: latest-1604-php5

MAINTAINER ESSch ESSchtolts@yandex.ru>

RUN cd / app / && (\

wget https://www.1c-bitrix.ru/download/small_business_encode.tar.gz \

&& tar -xf small_business_encode.tar.gz \

&& sed -i '5i php_value short_open_tag 1' .htaccess \

&& chmod -R 0777. \

&& sed -i 's / # php_value display_errors 1 / php_value display_errors 1 /' .htaccess \

&& sed -i '5i php_value opcache.revalidate_freq 0' .htaccess \

&& sed -i 's / # php_flag default_charset UTF-8 / php_flag default_charset UTF-8 /' .htaccess \

) && cd ..;

EXPOSE 80 3306

CMD ["/run.sh"]

esschtolts @ cloudshell: ~ / bitrix (essch) $ docker build -t essch / app: 0.12. | grep Successfully

Successfully built f76e656dac53

Successfully tagged essch / app: 0.12

esschtolts @ cloudshell: ~ / bitrix (essch) $ docker image push essch / app | grep digest

0.12: digest: sha256: 75c92396afacefdd5a3fb2024634a4c06e584e2a1674a866fa72f8430b19ff69 size: 11309

esschtolts @ cloudshell: ~ / bitrix (essch) $ cat deploymnet.yaml

apiVersion: apps / v1

kind: Deployment

metadata:

name: Nginxlamp

namespace: development

spec:

selector:

matchLabels:

app: lamp

replicas: 1

template:

metadata:

labels:

app: lamp

spec:

containers:

– name: lamp

image: essch / app: 0.12

ports:

– containerPort: 80

esschtolts @ cloudshell: ~ / bitrix (essch) $ IMAGE = essch / app: 0.12 kubectl create -f deploymnet.yaml

deployment.apps "Nginxlamp" created

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl get pods -l app = lamp

NAME READY STATUS RESTARTS AGE

Nginxlamp-55f8cd8dbc-mk9nk 1/1 Running 0 5m

esschtolts @ cloudshell: ~ / bitrix (essch) $ kubectl exec Nginxlamp-55f8cd8dbc-mk9nk – ls / app /

index.php

This happens because the developer of the images, which is correct and written in his documentation, expected that the image would be mounted to the host and the app folder was deleted in the script launched at the end. Also, in this approach, we will face the problem of constant updates of images, config (we cannot set the image number of a variable, since it will be executed on the cluster nodes) and container updates, we also cannot update the folder, since when the container is recreated, the changes will be returned to the original state.

The correct solution would be to mount the folder and include in the POD lifecycle the launch of the container, which starts in front of the main container and performs preparatory environment operations, often downloading the application from the repository, building, running tests, creating users and setting rights. For each operation, it is correct to launch a separate init container, in which this operation is the basic process, which are executed sequentially – by a chain that will be broken if one of the operations is performed with an error (it will return a non-zero process termination code). For such a container, a separate description is provided in the POD – InitContainer , listing them sequentially, they will build a chain of init container launches in the same order. In our case, we created an unnamed volume and, using InitContainer, delivered the installation files to it. After the successful completion of InitContainer , of which there may be several, the main one starts. The main container is already mounted in our volume, which already has the installation files, we just need to go to the browser and complete the installation:

esschtolts @ cloudshell: ~ / bitrix (essch) $ cat deploymnet.yaml

kind: Deployment

metadata:

name: Nginxlamp

namespace: development

spec:

selector:

matchLabels:

app: lamp

replicas: 1

template:

metadata:

labels:

app: lamp

spec:

initContainers:

– name: init

image: ubuntu

command:

– / bin / bash

– -c

– |

cd / app

apt-get update && apt-get install -y wget

wget https://www.1c-bitrix.ru/download/small_business_encode.tar.gz

tar -xf small_business_encode.tar.gz

sed -i '5i php_value short_open_tag 1' .htaccess

chmod -R 0777.

sed -i 's / # php_value display_errors 1 / php_value display_errors 1 /' .htaccess

sed -i '5i php_value opcache.revalidate_freq 0' .htaccess

sed -i 's / # php_flag default_charset UTF-8 / php_flag default_charset UTF-8 /' .htaccess

volumeMounts:

– name: app

mountPath: / app

containers:

– name: lamp

image: essch / app: 0.12

ports:

– containerPort: 80

volumeMounts:

– name: app

mountPath: / app

volumes:

– name: app

emptyDir: {}

You can watch events during POD creation with the watch kubectl get events command , and kubectl logs {ID_CONTAINER} -c init or more universally:

kubectl logs $ (kubectl get PODs -l app = lamp -o JSON | jq ".items [0] .metadata.name" | sed 's / "// g') -c init

It is advisable to choose small images for single tasks, for example, alpine: 3.5 :

esschtolts @ cloudshell: ~ (essch) $ docker pull alpine 1> \ dev \ null

esschtolts @ cloudshell: ~ (essch) $ docker pull ubuntu 1> \ dev \ null

esschtolts @ cloudshell: ~ (essch) $ docker images

REPOSITORY TAGIMAGE ID CREATED SIZE

ubuntu latest 93fd78260bd1 4 weeks ago 86.2MB

alpine latest 196d12cf6ab1 3 months ago 4.41MB

By slightly changing the code, we significantly saved on the size of the image:

image: alpine: 3.5

command:

– / bin / bash

– -c

– |

cd / app

apk –update add wget && rm -rf / var / cache / apk / *

tar -xf small_business_encode.tar.gz

rm -f small_business_encode.tar.gz

sed -i '5i php_value short_open_tag 1' .htaccess

sed -i 's / # php_value display_errors 1 / php_value display_errors 1 /' .htaccess

sed -i '5i php_value opcache.revalidate_freq 0' .htaccess

sed -i 's / # php_flag default_charset UTF-8 / php_flag default_charset UTF-8 /' .htaccess

chmod -R 0777.

volumeMounts:

There are also minimalistic images with pre-installed packages such as APIne with git: axeclbr / git and golang: 1-alpine .

Ways to ensure fault tolerance

Any process can crash. In the case of a container, if the main process crashes, then the container containing it also crashes. It is normal for the crash to occur during graceful shutdown. For example, our application in the container makes a backup of the database, in this case, after the container is executed, we get the work done. For demonstration purposes, let's take the sleep command:

vagrant @ ubuntu: ~ $ sudo docker pull ubuntu> / dev / null

vagrant @ ubuntu: ~ $ sudo docker run -d ubuntu sleep 60

0bd80651c6f97167b27f4e8df675780a14bd9e0a5c3f8e5e8340a98fc351bc64

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES

0bd80651c6f9 ubuntu "sleep 60" 15 seconds ago Up 12 seconds distracted_kalam

vagrant @ ubuntu: ~ $ sleep 60

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES

vagrant @ ubuntu: ~ $ sudo docker ps -a | grep ubuntu

0bd80651c6f9 ubuntu "sleep 60" 4 minutes ago Exited (0) 3 minutes ago distracted_kalam

In the case of backups, this is the norm, but in the case of applications that should not be terminated, it is not. A typical trick is a web server. The easiest thing in this case is to restart it:

vagrant @ ubuntu: ~ $ sudo docker run -d –restart = always ubuntu sleep 10

c3bc2d2e37a68636080898417f5b7468adc73a022588ae073bdb3a5bba270165

vagrant @ ubuntu: ~ $ sleep 30

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS

c3bc2d2e37a6 ubuntu sleep 10 "46 seconds ago Up 1 second

We see that when the container falls, it restarts. As a result – we always have an application in two states – raised or raised. If a web server crashes from some rare error, this is the norm, but most likely there is an error in processing requests, and it will crash on every such request, and in monitoring we will see a raised container. Such a web server is better dead than half alive. But, at the same time, a normal web server may not start due to rare errors, for example, due to the lack of connection to the database due to network instability. In such a case, the application must be able to handle errors and exit. And in case of a crash due to code errors, do not restart to see the inoperability and send it to the developers for repair. In the case of a floating error, you can try several times:

vagrant @ ubuntu: ~ $ sudo docker run -d –restart = on-failure: 3 ubuntu sleep 10

056c4fc6986a13936e5270585e0dc1782a9246f01d6243dd247cb03b7789de1c

vagrant @ ubuntu: ~ $ sleep 10

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

c3bc2d2e37a6 ubuntu "sleep 10" 9 minutes ago Up 2 seconds keen_sinoussi

vagrant @ ubuntu: ~ $ sleep 10

vagrant @ ubuntu: ~ $ sleep 10

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

c3bc2d2e37a6 ubuntu "sleep 10" 10 minutes ago Up 9 seconds keen_sinoussi

vagrant @ ubuntu: ~ $ sleep 10

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

c3bc2d2e37a6 ubuntu "sleep 10" 10 minutes ago Up 2 seconds keen_sinoussi

Another aspect is when to consider the container dead. By default, this is a process crash. But, by far, the application does not always crash itself in case of an error in order to allow the container to be restarted. For example, a server may be designed incorrectly and try to download the necessary libraries during its startup, but it does not have this opportunity, for example, due to the blocking of requests by the firewall. In such a scenario, the server can wait a long time if an adequate timeout is not specified. In this case, we need to check the functionality. For a web server, this is a response to a specific url, for example:

docker run –rm -d \

–-name = elasticsearch \

–-health-cmd = "curl –silent –fail localhost: 9200 / _cluster / health || exit 1" \

–-health-interval = 5s \

–-health-retries = 12 \

–-health-timeout = 20s \

{image}

For demonstration, we will use the file creation command. If the application has not reached the working state within the allotted time limit (set to 0) (for example, creating a file), then it is marked as working, but before that the specified number of checks is done:

vagrant @ ubuntu: ~ $ sudo docker run \

–d –name healt \

–-health-timeout = 0s \

–-health-interval = 5s \

–-health-retries = 3 \

–-health-cmd = "ls / halth" \

ubuntu bash -c 'sleep 1000'

c0041a8d973e74fe8c96a81b6f48f96756002485c74e51a1bd4b3bc9be0d9ec5

vagrant @ ubuntu: ~ $ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

c0041a8d973e ubuntu "bash -c 'sleep 1000'" 4 seconds ago Up 3 seconds (health: starting) healt

vagrant @ ubuntu: ~ $ sleep 20

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

c0041a8d973e ubuntu "bash -c 'sleep 1000'" 38 seconds ago Up 37 seconds (unhealthy) healt

vagrant @ ubuntu: ~ $ sudo docker rm -f healt

healt

If at least one of the checks worked, then the container is marked as healthy immediately:

vagrant @ ubuntu: ~ $ sudo docker run \

–d –name healt \

–-health-timeout = 0s \

–-health-interval = 5s \

–-health-retries = 3 \

–-health-cmd = "ls / halth" \

ubuntu bash -c 'touch / halth && sleep 1000'

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

160820d11933 ubuntu "bash -c 'touch / hal …" 4 seconds ago Up 2 seconds (health: starting) healt

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

160820d11933 ubuntu "bash -c 'touch / hal …" 6 seconds ago Up 5 seconds (healthy) healt

vagrant @ ubuntu: ~ $ sudo docker rm -f healt

healt

In this case, the checks are repeated all the time at a given interval:

vagrant @ ubuntu: ~ $ sudo docker run \

–d –name healt \

–-health-timeout = 0s \

–-health-interval = 5s \

–-health-retries = 3 \

–-health-cmd = "ls / halth" \

ubuntu bash -c 'touch / halth && sleep 60 && rm -f / halth && sleep 60'

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

8ec3a4abf74b ubuntu "bash -c 'touch / hal …" 7 seconds ago Up 5 seconds (health: starting) healt

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

8ec3a4abf74b ubuntu "bash -c 'touch / hal …" 24 seconds ago Up 22 seconds (healthy) healt

vagrant @ ubuntu: ~ $ sleep 60

vagrant @ ubuntu: ~ $ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

8ec3a4abf74b ubuntu "bash -c 'touch / hal …" About a minute ago Up About a minute (unhealthy) healt

Kubernetes provides (kubernetes.io/docs/tasks/configure-POD-container/configure-liveness-readiness-probes/) three tools that check the state of a container from outside. They are more important because they serve not only to inform, but also to manage the application life cycle, roll-forward and rollback of updates. Configuring them incorrectly can, and often does, cause the application to malfunction. So, if the liveness test is triggered before the application starts working, Kubernetes will kill the container, not allowing it to rise. Let's consider it in more detail. The liveness probe is used to determine the health of the application, and if the application crashes and does not respond to the liveness probe, Kubernetes reloads the container. As an example, we will take a shell test, due to the simplicity of the demonstration of work, but in practice it should be used only in extreme cases, for example, if the container is started not as a long-lived server, but as a JOB, doing its job and ending its existence, having achieved the result … For server checks, it is better to use HTTP probes, which already have a built-in dedicated proxy and do not require curl in the container and do not depend on external kube-proxy settings. When using databases, you must use a TCP probe, as they usually do not support the HTTP protocol. Let's create a long-lived container at www.katacoda.com/courses/kubernetes/playground:

controlplane $ cat << EOF> liveness.yaml

apiVersion: v1

kind: Pod

metadata:

name: liveness

spec:

containers:

– name: healtcheck

image: alpine: 3.5

args:

– / bin / sh

– -c

– touch / tmp / healthy; sleep 10; rm -rf / tmp / healthy; sleep 60

livenessProbe:

exec:

command:

– cat

– / tmp / healthy

initialDelaySeconds: 15

periodSeconds: 5

EOF

controlplane $ kubectl create -f liveness.yaml

pod / liveness created

controlplane $ kubectl get pods

NAME READY STATUS RESTARTS AGE

liveness 1/1 Running 2 2m11s

controlplane $ kubectl describe pod / liveness | tail -n 10

Type Reason Age From Message

–– – – – –

Normal Scheduled 2m37s default-scheduler Successfully assigned default / liveness to node01

Normal Pulling 2m33s kubelet, node01 Pulling image "alpine: 3.5"

Normal Pulled 2m30s kubelet, node01 Successfully pulled image "alpine: 3.5"

Normal Created 33s (x3 over 2m30s) kubelet, node01 Created container healtcheck

Normal Started 33s (x3 over 2m30s) kubelet, node01 Started container healtcheck

Normal Pulled 33s (x2 over 93s) kubelet, node01 Container image "alpine: 3.5" already present on machine

Warning Unhealthy 3s (x9 over 2m13s) kubelet, node01 Liveness probe failed: cat: can't open '/ tmp / healthy': No such file or directory

Normal Killing 3s (x3 over 2m3s) kubelet, node01 Container healtcheck failed liveness probe, will be restarted

We see that the container is constantly being restarted.

controlplane $ cat << EOF> liveness.yaml

apiVersion: v1

kind: Pod

metadata:

name: liveness

spec:

containers:

– name: healtcheck

image: alpine: 3.5

args:

– / bin / sh

– -c

– touch / tmp / healthy; sleep 30; rm -rf / tmp / healthy; sleep 60

livenessProbe:

exec:

command:

– cat

– / tmp / healthy

initialDelaySeconds: 15

periodSeconds: 5

EOF

controlplane $ kubectl create -f liveness.yaml