User Login Failure
KubeSphere automatically creates a default user (admin/P@88w0rd
) when it is installed. A user cannot be used for login if the status is not Active or you use an incorrect password.
Here are some of the frequently asked questions about user login failure.
User Not Active
You may see an image below when the login fails. To find out the reason and solve the issue, perform the following steps:
Execute the following command to check the status of the user.
$ kubectl get users NAME EMAIL STATUS admin admin@kubesphere.io Active
Verify that
ks-controller-manager
is running and check if exceptions are contained in logs:kubectl -n kubesphere-system logs -l app=ks-controller-manager
Here are some possible reasons for this issue.
Admission webhooks malfunction in Kubernetes 1.19
Kubernetes 1.19 uses Golang 1.15 in coding, requiring the certificate for admission webhooks to be updated. This causes the failure of ks-controller
admission webhook.
Related error logs:
Internal error occurred: failed calling webhook "validating-user.kubesphere.io": Post "https://ks-controller-manager.kubesphere-system.svc:443/validate-email-iam-kubesphere-io-v1alpha2-user?timeout=30s": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
For more information about the issue and solution, see this GitHub issue.
ks-controller-manager malfunctions
ks-controller-manager
relies on two stateful Services: OpenLDAP and Jenkins. When OpenLDAP or Jenkins goes down, ks-controller-manager
will be in the status of reconcile
.
Execute the following commands to verify that OpenLDAP and Jenkins are running normally.
kubectl -n kubesphere-devops-system get po | grep -v Running
kubectl -n kubesphere-system get po | grep -v Running
kubectl -n kubesphere-system logs -l app=openldap
Related error logs:
failed to connect to ldap service, please check ldap status, error: factory is not able to fill the pool: LDAP Result Code 200 \"Network Error\": dial tcp: lookup openldap.kubesphere-system.svc on 169.254.25.10:53: no such host
Internal error occurred: failed calling webhook “validating-user.kubesphere.io”: Post https://ks-controller-manager.kubesphere-system.svc:443/validate-email-iam-kubesphere-io-v1alpha2-user?timeout=4s: context deadline exceeded
Solution
You need to restore OpenLDAP and Jenkins with good network connection, and then restart ks-controller-manager
.
kubectl -n kubesphere-system rollout restart deploy ks-controller-manager
Wrong code branch used
If you used the incorrect version of ks-installer, the versions of different components would not match after the installation. Execute the following commands to check version consistency. Note that the correct image tag is v3.4.1
.
kubectl -n kubesphere-system get deploy ks-installer -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl -n kubesphere-system get deploy ks-apiserver -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl -n kubesphere-system get deploy ks-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}'
Wrong Username or Password
Run the following command to verify that the username and the password are correct.
curl -u <USERNAME>:<PASSWORD> "http://`kubectl -n kubesphere-system get svc ks-apiserver -o jsonpath='{.spec.clusterIP}'`/api/v1/nodes"
Redis failure
ks-console
and ks-apiserver
use Redis to share data across multiple copies. Use the following commands to verify that Redis is running normally.
kubectl -n kubesphere-system logs -l app=ks-console
kubectl -n kubesphere-system get po | grep -v Running
# High Availability
kubectl -n kubesphere-system exec -it redis-ha-server-0 redis-cli info replication
kubectl -n kubesphere-system exec -it redis-ha-server-0 -- sh -c 'for i in `seq 0 2`; do nc -vz redis-ha-server-$i.redis-ha.kubesphere-system.svc 6379; done'
kubectl -n kubesphere-system logs -l app=redis-ha-haproxy
kubectl -n kubesphere-system logs -l app=redis-ha
# Single Replica
kubectl -n kubesphere-system logs -l app=redis
Related error logs:
1344:C 17 Sep 2020 17:13:18.099 # Failed opening the RDB file dump.rdb (in server root dir /data) for saving: Stale file handle
1:M 17 Sep 2020 17:13:18.198 # Background saving error
1:M 17 Sep 2020 17:13:24.014 * 1 changes in 3600 seconds. Saving...
1:M 17 Sep 2020 17:13:24.015 * Background saving started by pid 1345
1345:C 17 Sep 2020 17:13:24.016 # Failed opening the RDB file dump.rdb (in server root dir /data) for saving: Stale file handle
1:M 17 Sep 2020 17:13:24.115 # Background saving error
E0909 07:05:22.770468 1 redis.go:51] unable to reach redis host EOF
[WARNING] 252/094143 (6) : Server check_if_redis_is_master_0/R0 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.223.2.232')", check duration: 1000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 252/094143 (6) : Server check_if_redis_is_master_0/R1 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.223.2.232')", check duration: 1000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 252/094143 (6) : Server check_if_redis_is_master_0/R2 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.223.2.232')", check duration: 1000ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 252/094143 (6) : backend 'check_if_redis_is_master_0' has no server available!
Solution
You need to restore Redis and make sure it is running normally with good network connection between Pods. After that, restart ks-console
to synchronize the data across copies.
kubectl -n kubesphere-system rollout restart deploy ks-console
Feedback
Was this page Helpful?
Receive the latest news, articles and updates from KubeSphere
Thanks for the feedback. If you have a specific question about how to use KubeSphere, ask it on Slack. Open an issue in the GitHub repo if you want to report a problem or suggest an improvement.