ofilter/Istio_Examples
1
0
Fork 0
Code Issues 26 Pull Requests Packages Projects 1 Releases Wiki Activity

Did the TLS gateway example.

Browse Source 
Somewhat filled the troubleshooting documentation for a bit.
This commit is contained in:
savagebidoof
2023-04-23 07:09:22 +02:00
parent dd3dcdbdf3
commit ee0c95e56d
9 changed files with 375 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files

280
Istio/00-Troubleshooting/README.md
View File

@ -1,142 +1,127 @@
IDK put some text in there
# Istioctl analyze
### Start the packet capture process on the istio-proxy from a pod.
`istioctl analyze` reviews the current configuration set.
Can be helpful to spot some improvements on the current configurations set, as well of the possibility of displaying misconfigurations / lack of them that might be causing issues.
```shell
istioctl analyze
```
```text
✔ No validation issues found when analyzing namespace: default.
```
By using the flag -A, it will review from all namespaces
```shell
istioctl analyze -A
```
```text
Info [IST0102] (Namespace istio-operator) The namespace is not enabled for Istio injection. Run 'kubectl label namespace istio-operator istio-injection=enabled' to enable it, or 'kubectl label namespace istio-operator istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0118] (Service istio-system/grafana) Port name service (port: 3000, targetPort: 3000) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service istio-system/jaeger-collector) Port name jaeger-collector-grpc (port: 14250, targetPort: 14250) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service istio-system/jaeger-collector) Port name jaeger-collector-http (port: 14268, targetPort: 14268) doesn't follow the naming convention of Istio port.
```
One can specify/target a single namespace by using the flag `-n`
```shell
istioctl analyze -n istio-operator
```
```text
Info [IST0102] (Namespace istio-operator) The namespace is not enabled for Istio injection. Run 'kubectl label namespace istio-operator istio-injection=enabled' to enable it, or 'kubectl label namespace istio-operator istio-injection=disabled' to explicitly mark it as not needing injection.
```
# Start the packet capture process on the istio-proxy container from a pod.
Target a pod and start a packet capture on the istio-proxy container.
```shell
$ kubectl exec -n default "$(kubectl get pod -n default -l app=helloworld -o jsonpath={.items..metadata.name})" -c istio-proxy -- sudo tcpdump dst port 80 -A
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
...
```
### Logs
# Logs
Istio system logs
> **Note:**\
> Remember that you can use the command `watch` or `watch -n 5` (where 5 refers every 5 seconds) in case of being interested on execute this commands periodically.
## Istiod
```shell
kubectl logs -f deployments/istiod -n istio-system
kubectl logs -n istio-system -f deployments/istiod
```
## Ingress
## Istioctl proxy-config
The service targeted, `istio-ingressgateway`, is an Ingress Load Balancer service from Istio.
```shell
istioctl proxy-config all helloworld-nginx-5d99f88767-cwcmd
kubectl logs -n istio-system services/istio-ingressgateway
```
```txt
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
80 - inbound ORIGINAL_DST
BlackHoleCluster - - - STATIC
InboundPassthroughClusterIpv4 - - - ORIGINAL_DST
PassthroughCluster - - - ORIGINAL_DST
agent - - - STATIC
byeworld.foo.svc.cluster.local 9090 - outbound EDS
grafana.istio-system.svc.cluster.local 3000 - outbound EDS
helloworld.default.svc.cluster.local 8080 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 80 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15021 - outbound EDS
istiod.istio-system.svc.cluster.local 443 - outbound EDS
istiod.istio-system.svc.cluster.local 15010 - outbound EDS
istiod.istio-system.svc.cluster.local 15012 - outbound EDS
istiod.istio-system.svc.cluster.local 15014 - outbound EDS
jaeger-collector.istio-system.svc.cluster.local 9411 - outbound EDS
jaeger-collector.istio-system.svc.cluster.local 14250 - outbound EDS
jaeger-collector.istio-system.svc.cluster.local 14268 - outbound EDS
kiali.istio-system.svc.cluster.local 9090 - outbound EDS
kiali.istio-system.svc.cluster.local 20001 - outbound EDS
kube-dns.kube-system.svc.cluster.local 53 - outbound EDS
kube-dns.kube-system.svc.cluster.local 9153 - outbound EDS
kubernetes.default.svc.cluster.local 443 - outbound EDS
myistio-ingressgateway.istio-ingress.svc.cluster.local 80 - outbound EDS
myistio-ingressgateway.istio-ingress.svc.cluster.local 443 - outbound EDS
myistio-ingressgateway.istio-ingress.svc.cluster.local 15021 - outbound EDS
prometheus.istio-system.svc.cluster.local 9090 - outbound EDS
prometheus_stats - - - STATIC
sds-grpc - - - STATIC
tracing.istio-system.svc.cluster.local 80 - outbound EDS
tracing.istio-system.svc.cluster.local 16685 - outbound EDS
xds-grpc - - - STATIC
zipkin - - - STRICT_DNS
zipkin.istio-system.svc.cluster.local 9411 - outbound EDS
#### Invalid TLS context has neither subject CN nor SAN names
ADDRESS PORT MATCH DESTINATION
10.96.0.10 53 ALL Cluster: outbound|53||kube-dns.kube-system.svc.cluster.local
0.0.0.0 80 Trans: raw_buffer; App: http/1.1,h2c Route: 80
0.0.0.0 80 ALL PassthroughCluster
10.102.38.206 443 ALL Cluster: outbound|443||myistio-ingressgateway.istio-ingress.svc.cluster.local
10.109.184.232 443 ALL Cluster: outbound|443||istiod.istio-system.svc.cluster.local
10.96.0.1 443 ALL Cluster: outbound|443||kubernetes.default.svc.cluster.local
10.96.248.46 443 ALL Cluster: outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
10.98.124.246 3000 Trans: raw_buffer; App: http/1.1,h2c Route: grafana.istio-system.svc.cluster.local:3000
10.98.124.246 3000 ALL Cluster: outbound|3000||grafana.istio-system.svc.cluster.local
0.0.0.0 8080 Trans: raw_buffer; App: http/1.1,h2c Route: 8080
0.0.0.0 8080 ALL PassthroughCluster
0.0.0.0 9090 Trans: raw_buffer; App: http/1.1,h2c Route: 9090
0.0.0.0 9090 ALL PassthroughCluster
10.96.0.10 9153 Trans: raw_buffer; App: http/1.1,h2c Route: kube-dns.kube-system.svc.cluster.local:9153
10.96.0.10 9153 ALL Cluster: outbound|9153||kube-dns.kube-system.svc.cluster.local
0.0.0.0 9411 Trans: raw_buffer; App: http/1.1,h2c Route: 9411
0.0.0.0 9411 ALL PassthroughCluster
10.100.204.154 14250 Trans: raw_buffer; App: http/1.1,h2c Route: jaeger-collector.istio-system.svc.cluster.local:14250
10.100.204.154 14250 ALL Cluster: outbound|14250||jaeger-collector.istio-system.svc.cluster.local
10.100.204.154 14268 Trans: raw_buffer; App: http/1.1,h2c Route: jaeger-collector.istio-system.svc.cluster.local:14268
10.100.204.154 14268 ALL Cluster: outbound|14268||jaeger-collector.istio-system.svc.cluster.local
0.0.0.0 15001 ALL PassthroughCluster
0.0.0.0 15001 Addr: *:15001 Non-HTTP/Non-TCP
0.0.0.0 15006 Addr: *:15006 Non-HTTP/Non-TCP
0.0.0.0 15006 Trans: tls; App: istio-http/1.0,istio-http/1.1,istio-h2; Addr: 0.0.0.0/0 InboundPassthroughClusterIpv4
0.0.0.0 15006 Trans: tls; Addr: 0.0.0.0/0 InboundPassthroughClusterIpv4
0.0.0.0 15006 Trans: tls; Addr: *:80 Cluster: inbound|80||
0.0.0.0 15010 Trans: raw_buffer; App: http/1.1,h2c Route: 15010
0.0.0.0 15010 ALL PassthroughCluster
10.109.184.232 15012 ALL Cluster: outbound|15012||istiod.istio-system.svc.cluster.local
0.0.0.0 15014 Trans: raw_buffer; App: http/1.1,h2c Route: 15014
0.0.0.0 15014 ALL PassthroughCluster
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
10.102.38.206 15021 Trans: raw_buffer; App: http/1.1,h2c Route: myistio-ingressgateway.istio-ingress.svc.cluster.local:15021
10.102.38.206 15021 ALL Cluster: outbound|15021||myistio-ingressgateway.istio-ingress.svc.cluster.local
10.96.248.46 15021 Trans: raw_buffer; App: http/1.1,h2c Route: istio-ingressgateway.istio-system.svc.cluster.local:15021
10.96.248.46 15021 ALL Cluster: outbound|15021||istio-ingressgateway.istio-system.svc.cluster.local
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
0.0.0.0 16685 Trans: raw_buffer; App: http/1.1,h2c Route: 16685
0.0.0.0 16685 ALL PassthroughCluster
0.0.0.0 20001 Trans: raw_buffer; App: http/1.1,h2c Route: 20001
0.0.0.0 20001 ALL PassthroughCluster
The TLS certificate specified don't have the field CN or the field SAN.
NAME DOMAINS MATCH VIRTUAL SERVICE
myistio-ingressgateway.istio-ingress.svc.cluster.local:15021 * /*
8080 helloworld, helloworld.default + 1 more... /*
kube-dns.kube-system.svc.cluster.local:9153 * /*
80 istio-ingressgateway.istio-system, 10.96.248.46 /*
80 myistio-ingressgateway.istio-ingress, 10.102.38.206 /*
80 tracing.istio-system, 10.103.51.183 /*
jaeger-collector.istio-system.svc.cluster.local:14250 * /*
grafana.istio-system.svc.cluster.local:3000 * /*
istio-ingressgateway.istio-system.svc.cluster.local:15021 * /*
* /stats/prometheus*
InboundPassthroughClusterIpv4 * /*
* /healthz/ready*
inbound|80|| * /*
jaeger-collector.istio-system.svc.cluster.local:14268 * /*
9090 byeworld.foo, 10.103.187.190 /*
9090 kiali.istio-system, 10.104.141.120 /*
9090 prometheus.istio-system, 10.107.129.0 /*
9411 jaeger-collector.istio-system, 10.100.204.154 /*
9411 zipkin.istio-system, 10.104.238.43 /*
15010 istiod.istio-system, 10.109.184.232 /*
15014 istiod.istio-system, 10.109.184.232 /*
16685 tracing.istio-system, 10.103.51.183 /*
20001 kiali.istio-system, 10.104.141.120 /*
To address this issue, issue a new certificate that has at least one of those fields.
RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
default Cert Chain ACTIVE true 224526398421470636195992462181330755939 2023-04-23T23:57:50Z 2023-04-22T23:55:50Z
ROOTCA CA ACTIVE true 3144612513681150263454419199256531619 2033-04-17T19:15:16Z 2023-04-20T19:15:16Z
#### initial fetch timed out for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secretthread
This is due not being able to retrieve the TLS configuration assigned to the gateway.
It's Important that the secret is located in the same namespace as the Istio Load Balancer used. In my case is the `istio-system`, but it will vary based on the environment.
# Istioctl proxy-config
## Check listeners
Useful to review which is the configuration assigned to an Istio ingress. / Confirm if the configuration we are intending to deploy is being applied / learned.
### Get Istio ingress pod name
> **Note:**\
> Depending on the ingress gateway set, and your environment, it could be that the Load Balancer is not located in the namespace `istio-system`.
```shell
kubectl get pods -n istio-system
```
```text
NAME READY STATUS RESTARTS AGE
grafana-6cb5b7fbb8-2nlp6 1/1 Running 0 2d3h
istio-ingressgateway-864db96c47-nvjc7 1/1 Running 0 20h
istiod-649d466b9-bwx7j 1/1 Running 0 2d8h
jaeger-cc4688b98-h52xt 1/1 Running 0 2d3h
kiali-594965b98c-zc67p 1/1 Running 0 2d3h
prometheus-67f6764db9-szd5b 2/2 Running 0 2d3h
```
### List listeners
```shell
kubectl get pods -n istio-system istio-ingressgateway-864db96c47-nvjc7
```
```text
istioctl proxy-config listeners -n istio-system istio-ingressgateway-864db96c47-nvjc7
ADDRESS PORT MATCH DESTINATION
0.0.0.0 8443 SNI: lb.net Route: https.443.secure-http.helloworld-gateway.default
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
```
This makes reference to the configuration set in the gateway resources.
Here we can notice a route with SNI match "lb.net", which is listening to the port 443 and HTTPS protocol.
## Check logs verbosity level settings
`istioctl proxy-config log` will display the verbosity level set from each log type for the specified pod.
```shell
istioctl proxy-config log helloworld-nginx-5d99f88767-cwcmd
@ -153,53 +138,22 @@ active loggers:
client: warning
config: warning
connection: warning
conn_handler: warning
decompression: warning
dns: warning
dubbo: warning
envoy_bug: warning
ext_authz: warning
ext_proc: warning
rocketmq: warning
file: warning
filter: warning
forward_proxy: warning
grpc: warning
happy_eyeballs: warning
hc: warning
health_checker: warning
http: warning
http2: warning
hystrix: warning
init: warning
io: warning
jwt: warning
kafka: warning
key_value_store: warning
lua: warning
main: warning
matcher: warning
misc: error
mongo: warning
multi_connection: warning
oauth2: warning
quic: warning
quic_stream: warning
pool: warning
rate_limit_quota: warning
rbac: warning
rds: warning
redis: warning
router: warning
runtime: warning
stats: warning
secret: warning
tap: warning
testing: warning
thrift: warning
tracing: warning
upstream: warning
udp: warning
wasm: warning
websocket: warning
...
```
## List all
It displays ALL from the specified pod.
```shell
istioctl proxy-config all helloworld-nginx-5d99f88767-cwcmd
```
```txt
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
80 - inbound ORIGINAL_DST
BlackHoleCluster - - - STATIC
InboundPassthroughClusterIpv4 - - - ORIGINAL_DST
PassthroughCluster - - - ORIGINAL_DST
agent - - - STATIC
...
```