ofilter/Istio_Examples
1
0
Fork 0
Code Issues 26 Pull Requests Packages Projects 1 Releases Wiki Activity

Did the TLS gateway example.

Browse Source 
Somewhat filled the troubleshooting documentation for a bit.
This commit is contained in:
savagebidoof 2023-04-23 07:09:22 +02:00
parent dd3dcdbdf3
commit ee0c95e56d
9 changed files with 375 additions and 205 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

280
Istio/00-Troubleshooting/README.md
View File

@ -1,142 +1,127 @@
IDK put some text in there
# Istioctl analyze
### Start the packet capture process on the istio-proxy from a pod.
`istioctl analyze` reviews the current configuration set.
Can be helpful to spot some improvements on the current configurations set, as well of the possibility of displaying misconfigurations / lack of them that might be causing issues.
```shell
istioctl analyze
```
```text
✔ No validation issues found when analyzing namespace: default.
```
By using the flag -A, it will review from all namespaces
```shell
istioctl analyze -A
```
```text
Info [IST0102] (Namespace istio-operator) The namespace is not enabled for Istio injection. Run 'kubectl label namespace istio-operator istio-injection=enabled' to enable it, or 'kubectl label namespace istio-operator istio-injection=disabled' to explicitly mark it as not needing injection.
Info [IST0118] (Service istio-system/grafana) Port name service (port: 3000, targetPort: 3000) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service istio-system/jaeger-collector) Port name jaeger-collector-grpc (port: 14250, targetPort: 14250) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service istio-system/jaeger-collector) Port name jaeger-collector-http (port: 14268, targetPort: 14268) doesn't follow the naming convention of Istio port.
```
One can specify/target a single namespace by using the flag `-n`
```shell
istioctl analyze -n istio-operator
```
```text
Info [IST0102] (Namespace istio-operator) The namespace is not enabled for Istio injection. Run 'kubectl label namespace istio-operator istio-injection=enabled' to enable it, or 'kubectl label namespace istio-operator istio-injection=disabled' to explicitly mark it as not needing injection.
```
# Start the packet capture process on the istio-proxy container from a pod.
Target a pod and start a packet capture on the istio-proxy container.
```shell
$ kubectl exec -n default "$(kubectl get pod -n default -l app=helloworld -o jsonpath={.items..metadata.name})" -c istio-proxy -- sudo tcpdump dst port 80 -A
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
...
```
### Logs
# Logs
Istio system logs
> **Note:**\
> Remember that you can use the command `watch` or `watch -n 5` (where 5 refers every 5 seconds) in case of being interested on execute this commands periodically.
## Istiod
```shell
kubectl logs -f deployments/istiod -n istio-system
kubectl logs -n istio-system -f deployments/istiod
```
## Ingress
## Istioctl proxy-config
The service targeted, `istio-ingressgateway`, is an Ingress Load Balancer service from Istio.
```shell
istioctl proxy-config all helloworld-nginx-5d99f88767-cwcmd
kubectl logs -n istio-system services/istio-ingressgateway
```
```txt
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
80 - inbound ORIGINAL_DST
BlackHoleCluster - - - STATIC
InboundPassthroughClusterIpv4 - - - ORIGINAL_DST
PassthroughCluster - - - ORIGINAL_DST
agent - - - STATIC
byeworld.foo.svc.cluster.local 9090 - outbound EDS
grafana.istio-system.svc.cluster.local 3000 - outbound EDS
helloworld.default.svc.cluster.local 8080 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 80 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15021 - outbound EDS
istiod.istio-system.svc.cluster.local 443 - outbound EDS
istiod.istio-system.svc.cluster.local 15010 - outbound EDS
istiod.istio-system.svc.cluster.local 15012 - outbound EDS
istiod.istio-system.svc.cluster.local 15014 - outbound EDS
jaeger-collector.istio-system.svc.cluster.local 9411 - outbound EDS
jaeger-collector.istio-system.svc.cluster.local 14250 - outbound EDS
jaeger-collector.istio-system.svc.cluster.local 14268 - outbound EDS
kiali.istio-system.svc.cluster.local 9090 - outbound EDS
kiali.istio-system.svc.cluster.local 20001 - outbound EDS
kube-dns.kube-system.svc.cluster.local 53 - outbound EDS
kube-dns.kube-system.svc.cluster.local 9153 - outbound EDS
kubernetes.default.svc.cluster.local 443 - outbound EDS
myistio-ingressgateway.istio-ingress.svc.cluster.local 80 - outbound EDS
myistio-ingressgateway.istio-ingress.svc.cluster.local 443 - outbound EDS
myistio-ingressgateway.istio-ingress.svc.cluster.local 15021 - outbound EDS
prometheus.istio-system.svc.cluster.local 9090 - outbound EDS
prometheus_stats - - - STATIC
sds-grpc - - - STATIC
tracing.istio-system.svc.cluster.local 80 - outbound EDS
tracing.istio-system.svc.cluster.local 16685 - outbound EDS
xds-grpc - - - STATIC
zipkin - - - STRICT_DNS
zipkin.istio-system.svc.cluster.local 9411 - outbound EDS
#### Invalid TLS context has neither subject CN nor SAN names
ADDRESS PORT MATCH DESTINATION
10.96.0.10 53 ALL Cluster: outbound|53||kube-dns.kube-system.svc.cluster.local
0.0.0.0 80 Trans: raw_buffer; App: http/1.1,h2c Route: 80
0.0.0.0 80 ALL PassthroughCluster
10.102.38.206 443 ALL Cluster: outbound|443||myistio-ingressgateway.istio-ingress.svc.cluster.local
10.109.184.232 443 ALL Cluster: outbound|443||istiod.istio-system.svc.cluster.local
10.96.0.1 443 ALL Cluster: outbound|443||kubernetes.default.svc.cluster.local
10.96.248.46 443 ALL Cluster: outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
10.98.124.246 3000 Trans: raw_buffer; App: http/1.1,h2c Route: grafana.istio-system.svc.cluster.local:3000
10.98.124.246 3000 ALL Cluster: outbound|3000||grafana.istio-system.svc.cluster.local
0.0.0.0 8080 Trans: raw_buffer; App: http/1.1,h2c Route: 8080
0.0.0.0 8080 ALL PassthroughCluster
0.0.0.0 9090 Trans: raw_buffer; App: http/1.1,h2c Route: 9090
0.0.0.0 9090 ALL PassthroughCluster
10.96.0.10 9153 Trans: raw_buffer; App: http/1.1,h2c Route: kube-dns.kube-system.svc.cluster.local:9153
10.96.0.10 9153 ALL Cluster: outbound|9153||kube-dns.kube-system.svc.cluster.local
0.0.0.0 9411 Trans: raw_buffer; App: http/1.1,h2c Route: 9411
0.0.0.0 9411 ALL PassthroughCluster
10.100.204.154 14250 Trans: raw_buffer; App: http/1.1,h2c Route: jaeger-collector.istio-system.svc.cluster.local:14250
10.100.204.154 14250 ALL Cluster: outbound|14250||jaeger-collector.istio-system.svc.cluster.local
10.100.204.154 14268 Trans: raw_buffer; App: http/1.1,h2c Route: jaeger-collector.istio-system.svc.cluster.local:14268
10.100.204.154 14268 ALL Cluster: outbound|14268||jaeger-collector.istio-system.svc.cluster.local
0.0.0.0 15001 ALL PassthroughCluster
0.0.0.0 15001 Addr: *:15001 Non-HTTP/Non-TCP
0.0.0.0 15006 Addr: *:15006 Non-HTTP/Non-TCP
0.0.0.0 15006 Trans: tls; App: istio-http/1.0,istio-http/1.1,istio-h2; Addr: 0.0.0.0/0 InboundPassthroughClusterIpv4
0.0.0.0 15006 Trans: tls; Addr: 0.0.0.0/0 InboundPassthroughClusterIpv4
0.0.0.0 15006 Trans: tls; Addr: *:80 Cluster: inbound|80||
0.0.0.0 15010 Trans: raw_buffer; App: http/1.1,h2c Route: 15010
0.0.0.0 15010 ALL PassthroughCluster
10.109.184.232 15012 ALL Cluster: outbound|15012||istiod.istio-system.svc.cluster.local
0.0.0.0 15014 Trans: raw_buffer; App: http/1.1,h2c Route: 15014
0.0.0.0 15014 ALL PassthroughCluster
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
10.102.38.206 15021 Trans: raw_buffer; App: http/1.1,h2c Route: myistio-ingressgateway.istio-ingress.svc.cluster.local:15021
10.102.38.206 15021 ALL Cluster: outbound|15021||myistio-ingressgateway.istio-ingress.svc.cluster.local
10.96.248.46 15021 Trans: raw_buffer; App: http/1.1,h2c Route: istio-ingressgateway.istio-system.svc.cluster.local:15021
10.96.248.46 15021 ALL Cluster: outbound|15021||istio-ingressgateway.istio-system.svc.cluster.local
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
0.0.0.0 16685 Trans: raw_buffer; App: http/1.1,h2c Route: 16685
0.0.0.0 16685 ALL PassthroughCluster
0.0.0.0 20001 Trans: raw_buffer; App: http/1.1,h2c Route: 20001
0.0.0.0 20001 ALL PassthroughCluster
The TLS certificate specified don't have the field CN or the field SAN.
NAME DOMAINS MATCH VIRTUAL SERVICE
myistio-ingressgateway.istio-ingress.svc.cluster.local:15021 * /*
8080 helloworld, helloworld.default + 1 more... /*
kube-dns.kube-system.svc.cluster.local:9153 * /*
80 istio-ingressgateway.istio-system, 10.96.248.46 /*
80 myistio-ingressgateway.istio-ingress, 10.102.38.206 /*
80 tracing.istio-system, 10.103.51.183 /*
jaeger-collector.istio-system.svc.cluster.local:14250 * /*
grafana.istio-system.svc.cluster.local:3000 * /*
istio-ingressgateway.istio-system.svc.cluster.local:15021 * /*
* /stats/prometheus*
InboundPassthroughClusterIpv4 * /*
* /healthz/ready*
inbound|80|| * /*
jaeger-collector.istio-system.svc.cluster.local:14268 * /*
9090 byeworld.foo, 10.103.187.190 /*
9090 kiali.istio-system, 10.104.141.120 /*
9090 prometheus.istio-system, 10.107.129.0 /*
9411 jaeger-collector.istio-system, 10.100.204.154 /*
9411 zipkin.istio-system, 10.104.238.43 /*
15010 istiod.istio-system, 10.109.184.232 /*
15014 istiod.istio-system, 10.109.184.232 /*
16685 tracing.istio-system, 10.103.51.183 /*
20001 kiali.istio-system, 10.104.141.120 /*
To address this issue, issue a new certificate that has at least one of those fields.
RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
default Cert Chain ACTIVE true 224526398421470636195992462181330755939 2023-04-23T23:57:50Z 2023-04-22T23:55:50Z
ROOTCA CA ACTIVE true 3144612513681150263454419199256531619 2033-04-17T19:15:16Z 2023-04-20T19:15:16Z
#### initial fetch timed out for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secretthread
This is due not being able to retrieve the TLS configuration assigned to the gateway.
It's Important that the secret is located in the same namespace as the Istio Load Balancer used. In my case is the `istio-system`, but it will vary based on the environment.
# Istioctl proxy-config
## Check listeners
Useful to review which is the configuration assigned to an Istio ingress. / Confirm if the configuration we are intending to deploy is being applied / learned.
### Get Istio ingress pod name
> **Note:**\
> Depending on the ingress gateway set, and your environment, it could be that the Load Balancer is not located in the namespace `istio-system`.
```shell
kubectl get pods -n istio-system
```
```text
NAME READY STATUS RESTARTS AGE
grafana-6cb5b7fbb8-2nlp6 1/1 Running 0 2d3h
istio-ingressgateway-864db96c47-nvjc7 1/1 Running 0 20h
istiod-649d466b9-bwx7j 1/1 Running 0 2d8h
jaeger-cc4688b98-h52xt 1/1 Running 0 2d3h
kiali-594965b98c-zc67p 1/1 Running 0 2d3h
prometheus-67f6764db9-szd5b 2/2 Running 0 2d3h
```
### List listeners
```shell
kubectl get pods -n istio-system istio-ingressgateway-864db96c47-nvjc7
```
```text
istioctl proxy-config listeners -n istio-system istio-ingressgateway-864db96c47-nvjc7
ADDRESS PORT MATCH DESTINATION
0.0.0.0 8443 SNI: lb.net Route: https.443.secure-http.helloworld-gateway.default
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
```
This makes reference to the configuration set in the gateway resources.
Here we can notice a route with SNI match "lb.net", which is listening to the port 443 and HTTPS protocol.
## Check logs verbosity level settings
`istioctl proxy-config log` will display the verbosity level set from each log type for the specified pod.
```shell
istioctl proxy-config log helloworld-nginx-5d99f88767-cwcmd
@ -153,53 +138,22 @@ active loggers:
client: warning
config: warning
connection: warning
conn_handler: warning
decompression: warning
dns: warning
dubbo: warning
envoy_bug: warning
ext_authz: warning
ext_proc: warning
rocketmq: warning
file: warning
filter: warning
forward_proxy: warning
grpc: warning
happy_eyeballs: warning
hc: warning
health_checker: warning
http: warning
http2: warning
hystrix: warning
init: warning
io: warning
jwt: warning
kafka: warning
key_value_store: warning
lua: warning
main: warning
matcher: warning
misc: error
mongo: warning
multi_connection: warning
oauth2: warning
quic: warning
quic_stream: warning
pool: warning
rate_limit_quota: warning
rbac: warning
rds: warning
redis: warning
router: warning
runtime: warning
stats: warning
secret: warning
tap: warning
testing: warning
thrift: warning
tracing: warning
upstream: warning
udp: warning
wasm: warning
websocket: warning
...
```
## List all
It displays ALL from the specified pod.
```shell
istioctl proxy-config all helloworld-nginx-5d99f88767-cwcmd
```
```txt
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
80 - inbound ORIGINAL_DST
BlackHoleCluster - - - STATIC
InboundPassthroughClusterIpv4 - - - ORIGINAL_DST
PassthroughCluster - - - ORIGINAL_DST
agent - - - STATIC
...
```

174
Istio/02-Traffic_management/07-HTTPS-Gateway-Simple-TLS/README.md Normal file
View File

@ -0,0 +1,174 @@
# Based on
- 01-hello_world_1_service_1_deployment
# Description
On this example, we generate a TLS configuration, and afterwards we attach such to a `Gateway` resource listening to the port `443` for `HTTPS` traffic.
> **Note:** \
> This was based on the information from the following Istio documentation:
> - [Secure Gateways](https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/)
# Configuration applied
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: helloworld-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: secure-http
protocol: HTTPS
hosts:
- "*"
tls:
mode: SIMPLE
credentialName: my-tls-cert-secret
```
- Gateway is listening to the port `443` and `HTTPS` protocol.
- Allows for all hosts.
- The TLS configuration is set to simple, and the credentials (the object that contains the certificates/TLS configuration) is set to `my-tls-cert-secret`.
> **Note:**\
> The credentials resource is created further bellow through the [Walkthrough](#Walkthrough) steps.
> **Note:**\
> For more information regarding the TLS mode configuration, refer to the following [Istio documentation regarding the TLS mode field](https://istio.io/latest/docs/reference/config/networking/gateway/#ServerTLSSettings-TLSmode).
# Walkthrough
## Generate client and server certificate and key files
First step will be to generate the certificate and key files to be able to set them to the Gateway resource.
### Create a folder to store files.
Create the folder to contain the files that will be generated.
```shell
mkdir certfolder
```
### Create a certificate and a private key.
```shell
openssl req -x509 -sha256 -nodes -days 365 -subj '/O=Internet of things/CN=lb.net' -newkey rsa:2048 -keyout certfolder/istio.cert.key -out certfolder/istio.cert.crt
```
The files generated are the following:
```yaml
private-key: certfolder/istio.cert.key
root-certificate: certfolder/istio.cert.crt
```
The information set to the certificate generated is the following:
```yaml
Organization-name: Internet of things
CN: lb.net
```
### Create a TLS secret
At this step we create the tls secret `my-tls-cert-secret` on the namespace `istio-system`.
```shell
kubectl create -n istio-system secret tls my-tls-cert-secret \
--key=certfolder/istio.cert.key \
--cert=certfolder/istio.cert.crt
```
```text
secret/my-tls-cert-secret created
```
```text
service/helloworld created
deployment.apps/helloworld-nginx created
gateway.networking.istio.io/helloworld-gateway created
virtualservice.networking.istio.io/helloworld-vs created
```
> **Note:**\
> It's Important that the secret is located in the same namespace as the Load Balancer used. In my case is the `istio-system`, but it will vary based on the environment.
## Deploy resources
```shell
kubectl apply -f ./
```
```text
service/helloworld created
deployment.apps/helloworld-nginx created
gateway.networking.istio.io/helloworld-gateway created
virtualservice.networking.istio.io/helloworld-vs created
```
## Test the service
[//]: # (```shell)
[//]: # (curl --insecure --resolve lb.net:443:192.168.1.50 https://lb.net/helloworld)
[//]: # (```)
```shell
curl --insecure https://192.168.1.50/helloworld -I
```
```text
HTTP/2 200
server: istio-envoy
date: Sun, 23 Apr 2023 05:06:47 GMT
content-type: text/html
content-length: 615
last-modified: Tue, 28 Mar 2023 15:01:54 GMT
etag: "64230162-267"
accept-ranges: bytes
x-envoy-upstream-service-time: 96
```
## Cleanup
```shell
kubectl delete -n istio-system secret my-tls-cert-secret
```
```shell
kubectl delete -f ./
```
```shell
rm -rv certfolder/
```
# Troubleshooting.
## curl: (7) Failed to connect to 192.168.1.51 port 443 after 2 ms: Couldn't connect to server
- Ensure that the gateway is listening to the right port, in this case, the port 443.
- Refer to the troubleshooting documentation, specifically the `Logs>Ingress`. \
Check if it displays any log activity that could facilitate the troubleshooting / investigation.
## curl: (35) Recv failure: Connection reset by peer
- Refer to the troubleshooting documentation, specifically the `Logs>Ingress`. \
Check if it displays any log activity that could facilitate the troubleshooting / investigation.
## 404
Ensure the URL used to thest the connectivity, matches the host and path rules applied, both in the `Gateway` and `VirtualService` resources.
# Links of Interest
- https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress
- https://istio.io/latest/docs/reference/config/networking/gateway/#ServerTLSSettings-TLSmode

39
Istio/02-Traffic_management/07-HTTPS-Gateway-Simple-TLS/deployment.yaml Executable file
View File

@ -0,0 +1,39 @@
apiVersion: v1
kind: Service
metadata:
name: helloworld
labels:
app: helloworld
service: helloworld
spec:
ports:
- port: 80
name: http
selector:
app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-nginx
labels:
app: helloworld
spec:
replicas: 1
selector:
matchLabels:
app: helloworld
template:
metadata:
labels:
app: helloworld
spec:
containers:
- name: helloworld
image: nginx
resources:
requests:
cpu: "100m"
imagePullPolicy: IfNotPresent #Always
ports:
- containerPort: 80

38
Istio/02-Traffic_management/07-HTTPS-Gateway-Simple-TLS/gateway.yaml Executable file
View File

@ -0,0 +1,38 @@
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: helloworld-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: secure-http
protocol: HTTPS
hosts:
- "*"
tls:
mode: SIMPLE
credentialName: my-tls-cert-secret
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: helloworld-vs
spec:
hosts:
- "*"
gateways:
- helloworld-gateway
http:
- match:
- uri:
exact: /helloworld
route:
- destination:
host: helloworld
port:
number: 80
rewrite:
uri: "/"

18
Istio/02-Traffic_management/README.md
View File

@ -9,21 +9,3 @@ ALL NEEDS DOCUMENTATION
- 05a-FaultInjection-delay
- 05b-FaultInjection-abort
- 06-mTLS (would need some documentation review, mainly go over the differences respective to the template/prior configuration used)
# TODO
minimum TLS version:
https://istio.io/latest/docs/tasks/security/tls-configuration/workload-min-tls-version/
Should try to do a double Virtual Service chain
https://academy.tetrate.io/courses/take/istio-fundamentals/lessons/19068816-lab-2-observing-failure-injection
Circuit breaking
https://istio.io/latest/docs/tasks/traffic-management/circuit-breaking/

31
Istio/README.md
View File

@ -1,4 +1,9 @@
# Stuff
## Glossary
https://istio.io/latest/docs/reference/glossary/
@ -19,33 +24,11 @@ https://istio.io/latest/docs/ops/deployment/vm-architecture/
https://kubebyexample.com/learning-paths/istio/intro
# Notes for myself
My current issues:
- Understanding authentication
https://tetrate.io/blog/istio-how-to-enforce-egress-traffic-using-istios-authorization-policies/
Multiple Ingress
https://youtu.be/QIkryA8HnQ0
https://github.com/redkubes/otomi-core/blob/main/charts/team-ns/templates/istio-gateway.yaml
Internal and external authentication should be set together.
https://istio.io/latest/docs/ops/diagnostic-tools/proxy-cmd/
Using service accounts

0
Istio/__Ingress/01-namespace.yaml → Istio/__Ingress/01-Create-Istio-LoadBalancer/01-namespace.yaml
View File

0
Istio/__Ingress/README.md → Istio/__Ingress/01-Create-Istio-LoadBalancer/README.md
View File

0
Istio/__Ingress/ingress.yaml → Istio/__Ingress/01-Create-Istio-LoadBalancer/ingress.yaml
View File