918f480319
Also have documented the ingress example regarding installing a Istio Ingress Gateway Load Balancer.
Table of Contents
Logs
Note:
Remember that you can use the commandwatchorwatch -n 5(where 5 refers every 5 seconds) in case of being interested on execute this commands periodically.
Istiod
kubectl logs -n istio-system -f deployments/istiod
Istio-Proxy Pod
This will display the logs from a deployment while targeting the istio-proxy container from the targeted pod/deployment.
As well will attach the session to stream new logs. (-f --follow)
kubectl logs deployments/helloworld-default -f -c istio-proxy
[2023-05-15T00:42:03.699Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 10.111.90.232:8080 172.17.121.65:52006 - -
[2023-05-15T00:42:24.785Z] "HEAD / HTTP/1.1" 200 - via_upstream - "-" 0 0 2 1 "-" "curl/7.74.0" "c133cbf0-b57d-4fba-8f84-d683ab903399" "helloworld.default.svc.cluster.local" "172.17.121.65:80" inbound|80|| 127.0.0.6:51695 172.17.121.65:80 172.17.121.65:43786 outbound_.80_._.helloworld.default.svc.cluster.local default
[2023-05-15T00:42:24.784Z] "HEAD / HTTP/1.1" 200 - via_upstream - "-" 0 0 5 4 "-" "curl/7.74.0" "c133cbf0-b57d-4fba-8f84-d683ab903399" "helloworld.default.svc.cluster.local" "172.17.121.65:80" outbound|80||helloworld.default.svc.cluster.local 172.17.121.65:43786 10.111.90.232:80 172.17.121.65:57030 - default
[2023-05-15T00:43:23.209Z] "HEAD / HTTP/1.1" 200 - via_upstream - "-" 0 0 6 5 "-" "curl/7.74.0" "e1f0a2f3-93ff-4c41-8cb3-6d3a53fce065" "helloworld.foo.svc.cluster.local" "172.17.247.42:80" outbound|80||helloworld.foo.svc.cluster.local 172.17.121.65:55040 10.109.248.148:80 172.17.121.65:60520 - default
[2023-05-15T00:43:29.751Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 10.109.248.148:8080 172.17.121.65:40370 - -
[2023-05-15T00:43:31.979Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 10.109.248.148:8080 172.17.121.65:40402 - -
Ingress
The service targeted, istio-ingressgateway, is an Ingress Load Balancer service from Istio.
kubectl logs -n istio-system services/istio-ingressgateway
Invalid TLS context has neither subject CN nor SAN names
The TLS certificate specified don't have the field CN or the field SAN.
To address this issue, issue a new certificate that has at least one of those fields.
initial fetch timed out for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secretthread
This is due not being able to retrieve the TLS configuration assigned to the gateway.
It's Important that the secret is located in the same namespace as the Istio Load Balancer used. In my case is the istio-system, but it will vary based on the environment.
Istioctl analyze
istioctl analyze reviews the current configuration set.
Can be helpful to spot some improvements on the current configurations set, as well of the possibility of displaying misconfigurations / lack of them that might be causing issues.
istioctl analyze
✔ No validation issues found when analyzing namespace: default.
By using the flag -A, it will review from all namespaces
istioctl analyze -A
Info [IST0102] (Namespace istio-operator) The namespace is not enabled for Istio injection. Run 'kubectl label namespace istio-operator istio-injection=enabled' to enable it, or 'kubectl label namespace istio-operator istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0118] (Service istio-system/grafana) Port name service (port: 3000, targetPort: 3000) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service istio-system/jaeger-collector) Port name jaeger-collector-grpc (port: 14250, targetPort: 14250) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service istio-system/jaeger-collector) Port name jaeger-collector-http (port: 14268, targetPort: 14268) doesn't follow the naming convention of Istio port.
One can specify/target a single namespace by using the flag -n
istioctl analyze -n istio-operator
Info [IST0102] (Namespace istio-operator) The namespace is not enabled for Istio injection. Run 'kubectl label namespace istio-operator istio-injection=enabled' to enable it, or 'kubectl label namespace istio-operator istio-injection=disabled' to explicitly mark it as not needing injection.
Example of spotting a misconfiguration
In this example, I have configured the gateway to listen to a port that currently is not open in the Isito Load Balancer selected.
istioctl analyze
Warning [IST0104] (Gateway default/helloworld-gateway) The gateway refers to a port that is not exposed on the workload (pod selector istio=ingressgateway; port 81)
Start the packet capture process on the istio-proxy container from a pod.
Target a pod and start a packet capture on the istio-proxy container.
This step requires Istio to be installed with the flag values.global.proxy.privileged=true
This is very useful to confirm if the service is receiving any traffic, or which is the traffic received.
If mTLS is enabled and configured, the traffic received should be encrypted.
kubectl exec -n default "$(kubectl get pod -n default -l app=helloworld -o jsonpath={.items..metadata.name})" -c istio-proxy -- sudo tcpdump dst port 80 -A
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
...
Istioctl proxy-status
Note: Shorthand is
ps
istioctl ps
NAME CLUSTER CDS LDS EDS RDS ECDS ISTIOD VERSION
helloworld-6798765f88-ql26n.default Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-dbf5ff64-9kxxs 1.17.2
helloworld2-dc9cb5db6-m47x7.default Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-dbf5ff64-9kxxs 1.17.2
istio-egressgateway-676bf68b54-d28fn.istio-system Kubernetes SYNCED SYNCED SYNCED NOT SENT NOT SENT istiod-dbf5ff64-9kxxs 1.17.2
istio-ingressgateway-8d56c999d-nv7ph.istio-system Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-dbf5ff64-9kxxs 1.17.2
Istioctl proxy-config
Note: Shorthand is
pc
Check listeners
Useful to review which is the configuration assigned to an Istio ingress. / Confirm if the configuration we are intending to deploy is being applied / learned.
Get Istio ingress pod name
Note:
Depending on the ingress gateway set, and your environment, it could be that the Load Balancer is not located in the namespaceistio-system.
kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-6cb5b7fbb8-2nlp6 1/1 Running 0 2d3h
istio-ingressgateway-864db96c47-nvjc7 1/1 Running 0 20h
istiod-649d466b9-bwx7j 1/1 Running 0 2d8h
jaeger-cc4688b98-h52xt 1/1 Running 0 2d3h
kiali-594965b98c-zc67p 1/1 Running 0 2d3h
prometheus-67f6764db9-szd5b 2/2 Running 0 2d3h
List listeners
kubectl get pods -n istio-system istio-ingressgateway-864db96c47-nvjc7
istioctl proxy-config listeners -n istio-system istio-ingressgateway-864db96c47-nvjc7
ADDRESS PORT MATCH DESTINATION
0.0.0.0 8443 SNI: lb.net Route: https.443.secure-http.helloworld-gateway.default
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
This makes reference to the configuration set in the gateway resources. Here we can notice a route with SNI match "lb.net", which is listening to the port 443 and HTTPS protocol.
Check logs verbosity level settings
istioctl proxy-config log will display the verbosity level set from each log type for the specified pod.
istioctl proxy-config log helloworld-nginx-5d99f88767-cwcmd
helloworld-nginx-5d99f88767-cwcmd.default:
active loggers:
admin: warning
alternate_protocols_cache: warning
aws: warning
assert: warning
backtrace: warning
cache_filter: warning
client: warning
config: warning
connection: warning
...
List all
It displays ALL from the specified pod.
istioctl proxy-config all helloworld-nginx-5d99f88767-cwcmd
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
80 - inbound ORIGINAL_DST
BlackHoleCluster - - - STATIC
InboundPassthroughClusterIpv4 - - - ORIGINAL_DST
PassthroughCluster - - - ORIGINAL_DST
agent - - - STATIC
...