问题
These are pretty much the steps I have followed in order. Basically what is outlined in the documentation:
https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv
azure-storage-claim.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: test-app-sc
provisioner: kubernetes.io/azure-file
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=1000
- gid=1000
- mfsymlinks
- nobrl
- cache=none
parameters:
skuName: Standard_LRS
location: westus
azure-storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-app-storage
spec:
accessModes:
- ReadWriteMany
storageClassName: test-app-sc
resources:
requests:
storage: 15Gi
PVC is now setup.
Changed the mountPath per the Postgres image documentation:
PGDATA
This optional variable can be used to define another location - like a subdirectory - for the database files. The default is /var/lib/postgresql/data, but if the data volume you're using is a filesystem mountpoint (like with GCE persistent disks), Postgres initdb recommends a subdirectory (for example /var/lib/postgresql/data/pgdata ) be created to contain the data.
This is an environment variable that is not Docker specific. Because the variable is used by the postgres server binary (see the PostgreSQL docs), the entrypoint script takes it into account.
Based on that, I have my postgres.yaml setup like the following:
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres-deployment
spec:
replicas: 1
selector:
matchLabels:
component: postgres
template:
metadata:
labels:
component: postgres
spec:
containers:
- name: postgres
image: postgres:11-alpine
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
valueFrom:
secretKeyRef:
name: test-app-secrets
key: PGDATABASE
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: test-app-secrets
key: PGUSER
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: test-app-secrets
key: PGPASSWORD
- name: POSTGRES_INITDB_ARGS
value: "-A md5"
- name: PGDATA
value: /var/lib/postgresql-data
volumeMounts:
- name: test-app-storage
mountPath: /var/lib/postgresql-data
subPath: postgres-storage
volumes:
- name: test-app-storage
persistentVolumeClaim:
claimName: test-app-storage
---
apiVersion: v1
kind: Service
metadata:
name: postgres-cluster-ip-service
spec:
type: ClusterIP
selector:
component: postgres
ports:
- port: 1423
targetPort: 5432
You get the error:
chmod: changing permissions of '/var/lib/postgresql-data': Operation not permitted
So with either of that as the Dockerfile:
FROM postgres:11-alpine
EXPOSE 5432
RUN /bin/bash -c 'chmod 777 /var/lib/postgresql-data'
Or
FROM postgres:11-alpine
EXPOSE 5432
It doesn't really matter, you still get the same type of error by doing any the following:
...
- name: POSTGRES_INITDB_ARGS
value: "-A md5"
volumeMounts:
- name: test-app-storage
mountPath: /var/lib/postgresql-data
subPath: postgres-storage
volumes:
- name: test-app-storage
persistentVolumeClaim:
claimName: test-app-storage
...
Results in the following error:
The files belonging to this database system will be owned by user "postgres". This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english".
Data page checksums are disabled.
initdb: error: directory "/var/lib/postgresql-data" exists but is not empty If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql-data" or run initdb with an argument other than "/var/lib/postgresql-data".
Try this:
...
- name: POSTGRES_INITDB_ARGS
value: "-A md5"
volumeMounts:
- name: test-app-storage
mountPath: /var/lib/postgresql-data
subPath: postgres-storage
volumes:
- name: test-app-storage
persistentVolumeClaim:
claimName: test-app-storage
...
And it results in this:
chmod: changing permissions of '/var/lib/postgresql-data': Operation not permitted
Try this:
...
- name: POSTGRES_INITDB_ARGS
value: "-A md5"
value: "-D /var/lib/postgresql/data/pgdata"
volumeMounts:
- name: test-app-storage
mountPath: /var/lib/postgresql/data/pgdata
subPath: postgres-storage
volumes:
- name: test-app-storage
persistentVolumeClaim:
claimName: test-app-storage
...
And it results in this:
The files belonging to this database system will be owned by user "postgres". This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english".
Data page checksums are disabled.
initdb: error: could not change permissions of directory "/var/lib/postgresql/data/pgdata": Operation not permitted fixing permissions on existing directory /var/lib/postgresql/data/pgdata ...
So nothing seems to work that I've tried and following the documentation where I can.
Someone suggested to get rid of the volume mounts like so:
...
- name: POSTGRES_INITDB_ARGS
value: "-A md5"
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumes:
- name: test-app-storage
persistentVolumeClaim:
claimName: test-app-storage
...
Which, hey, that actually works! But doesn't persist data given it just uses the Pod storage so is pretty pointless:
And sure enough when you create a table in Postgres, destroy the Pod, and then redeploy it, of course the table is no longer there.
So more than likely I'm doing something wrong, but I've been following the documentation and seems like this should work.
Where are things going wrong?
EDIT: Permissions in Pod
Apparently it is a permissions issue that occurs when PGDATA is the same directory as mountPath. For example:
...
- name: PGDATA
value: /var/lib/postgresql-data
volumeMounts:
- name: test-app-storage
mountPath: /var/lib/postgresql-data
subPath: postgres-storage
...
or
...
# if PGDATA is not specified it defaults to /var/lib/postgresql/data
# - name: PGDATA
# value: /var/lib/postgresql-data
volumeMounts:
- name: test-app-storage
mountPath: /var/lib/postgresql/data
subPath: postgres-storage
...
Something like this where they do not match will create the Pod, but uses Pod storage which I obviously don't want:
# Thus /var/lib/postgresql/data
# - name: PGDATA
# value: /var/lib/postgresql-data
volumeMounts:
- name: test-app-storage
mountPath: /var/lib/postgresql-data
subPath: postgres-storage
Permissions ls -l looks like this:
$ ls -l
drwxr-xr-x 1 root root 4096 Feb 2 06:06 apt
drwxr-xr-x 1 root root 4096 Feb 2 06:07 dpkg
drwxr-xr-x 2 root root 4096 Feb 2 06:06 exim4
drwxr-xr-x 2 root root 4096 Aug 28 2018 logrotate
drwxr-xr-x 2 root root 4096 Nov 10 12:17 misc
drwxr-xr-x 2 root root 4096 Jan 30 00:00 pam
drwxr-xr-x 1 postgres postgres 4096 Feb 2 06:07 postgresql
drwxrwxrwx 2 1000 1000 0 Jan 31 21:46 postgresql-data
drwxr-xr-x 1 root root 4096 Jan 30 00:00 systemd
drwxr-xr-x 3 root root 4096 Feb 2 06:07 ucf
$ ls -l postgresql && ls -l postgresql/data && ls -l postgresql-data
total 4
drwx------ 19 postgres postgres 4096 Feb 5 23:28 data
total 124
drwx------ 6 postgres postgres 4096 Feb 5 23:28 base
drwx------ 2 postgres postgres 4096 Feb 5 23:29 global
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_commit_ts
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_dynshmem
-rw------- 1 postgres postgres 4281 Feb 5 23:28 pg_hba.conf
-rw------- 1 postgres postgres 1636 Feb 5 23:28 pg_ident.conf
drwx------ 4 postgres postgres 4096 Feb 5 23:33 pg_logical
drwx------ 4 postgres postgres 4096 Feb 5 23:28 pg_multixact
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_notify
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_replslot
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_serial
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_snapshots
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_stat
drwx------ 2 postgres postgres 4096 Feb 5 23:51 pg_stat_tmp
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_subtrans
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_tblspc
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_twophase
-rw------- 1 postgres postgres 3 Feb 5 23:28 PG_VERSION
drwx------ 3 postgres postgres 4096 Feb 5 23:28 pg_wal
drwx------ 2 postgres postgres 4096 Feb 5 23:28 pg_xact
-rw------- 1 postgres postgres 88 Feb 5 23:28 postgresql.auto.conf
-rw------- 1 postgres postgres 26588 Feb 5 23:28 postgresql.conf
-rw------- 1 postgres postgres 36 Feb 5 23:28 postmaster.opts
-rw------- 1 postgres postgres 94 Feb 5 23:28 postmaster.pid
total 0
The permissions for where it creates the data files is postgres. However, doing this, it doesn't map to Azure Files and the PVC. It just stays and is destroyed with the Pod.
I think what is happening is mountPath uses root and PGDATA uses postgres, and somehow mountPath is trying to use postgres???
Really, not sure and still lost as to how to resolve it.
EDIT2
Came across this answer:
https://stackoverflow.com/a/51203031/3123109
So added the following to mine:
- name: postgres
image: postgres
command:
- /bin/chown
- -R
- "1000"
- /var/lib/postgresql/data
But this generates a new error:
The selected container has not logged any messages yet.
Progress, I guess.
回答1:
I think your problem might be due to the fact that you are trying to use a subPath as part of your deployment object
Please retry your first configuration without this line:
subPath: postgres-storage
It should result into this volumeMounts: - name: postgres-storage mountPath: /var/lib/postgresql/data/pgdata
Let me know if that helps.
UPDATE: The docker image for the postgresql requires special attention when using persistent storage
https://hub.docker.com/_/postgres
The main caveat to note is that postgres doesn't care what UID it runs as (as long as the owner of /var/lib/postgresql/data matches), but initdb does care (and needs the user to exist in /etc/passwd):
The three easiest ways to get around this:
use the Debian variants (not the Alpine variants) and thus allow the image to use the nss_wrapper library to "fake" /etc/passwd contents for you (see docker-library/postgres#448 for more details)
bind-mount /etc/passwd read-only from the host (if the UID you desire is a valid user on your host):
$ docker run -it --rm --user "$(id -u):$(id -g)" -v /etc/passwd:/etc/passwd:ro -e POSTGRES_PASSWORD=mysecretpassword postgres The files belonging to this database system will be owned by user "jsmith"
- Initialize the target directory separately from the final runtime (with a chown in between):
$ docker volume create pgdata $ docker run -it --rm -v pgdata:/var/lib/postgresql/data -e POSTGRES_PASSWORD=mysecretpassword postgres The files belonging to this database system will be owned by user "postgres". ( once it's finished initializing successfully and is waiting for connections, stop it ) $ docker run -it --rm -v pgdata:/var/lib/postgresql/data bash chown -R 1000:1000 /var/lib/postgresql/data $ docker run -it --rm --user 1000:1000 -v pgdata:/var/lib/postgresql/data postgres LOG: database system was shut down at 2017-01-20 00:03:23 UTC LOG: MultiXact member wraparound protections are now enabled LOG: autovacuum launcher started LOG: database system is ready to accept connections
One solution would be to leverage the helm chart made by bitnami, they have worked out the default of the complex persistent storage configurations using an init container, and they also support the /dev/shm configuration needed to fix
Also note that the default /dev/shm size for containers is 64MB. If the shared memory is exhausted you will encounter ERROR: could not resize shared memory segment . . . : No space left on device. You will want to pass --shm-size=256MB for example to docker run, or alternatively in docker-compose
Here's the example of the init container used by the chart.
https://github.com/helm/charts/blob/master/stable/postgresql/templates/statefulset.yaml#L74-L115
The recommendation would be to use helm to install the Postgres chart (which uses a statefuset) instead of trying to deploy the DB using a deployment object. (normally you want to use deployment objects for stateless application).
来源：https://stackoverflow.com/questions/60192819/cant-get-either-postgres-permissions-or-pvc-working-in-aks