Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: troubleshooting cluster #16

Open
wants to merge 3 commits into
base: cluster
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions cmd/cluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -374,10 +374,10 @@ func installGitServer() {
"--name=git-server",
"-v"+clusterRootPath+":/git-server/repos",
"-v"+clusterRootPath+"/.ssh:/git-server/keys",
"jkarlos/git-server-docker",
)
"jkarlos/git-server-docker")

err := cmd.Run()

if err != nil {
log.Fatalf("failed to start git server: %v", err)
}
Expand Down Expand Up @@ -464,7 +464,8 @@ func installSealedSecrets() {
}

// reload secret if exists
secretsPath := filepath.Join(clusterRootPath, ".secrets", "sealed-secrets.yaml")
secretsDir := filepath.Join(clusterRootPath, ".secrets")
secretsPath := filepath.Join(secretsDir, "sealed-secrets.yaml")
if _, err = os.Stat(secretsPath); os.IsExist(err) {
secret, err := ioutil.ReadFile(secretsPath)
check(err)
Expand Down Expand Up @@ -495,6 +496,8 @@ func installSealedSecrets() {
secret, err := cmd.Output()
check(err)

os.MkdirAll(secretsDir, 0755) //if .secrets folder is missing, create
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keep comments above the code they relate to, not inline.

also since there is no if statement here. perhaps to show the true intention of the code its better to say ensure secrets folder is present. or something along those lines.

log.Info("Trying to write new file")
err = ioutil.WriteFile(secretsPath, secret, 0644)
check(err)
}
Expand Down Expand Up @@ -561,7 +564,7 @@ func installFluxHelmOperator() {

err := cmd.Run()
if err != nil {
log.Fatal("failed to start flux helm operator: %v", err)
log.Fatalf("failed to start flux helm operator: %v", err)
}
}

Expand Down
2 changes: 1 addition & 1 deletion docs/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ git commit -m "modified podinfo deployment"
3. Within the context of the deploy folder, preform a git add and commit, eg.
```sh
git add -A
git commit -m "modified podinfo deployment"
git commit -m "removed podinfo deployment"
```
4. Your changes will take effect in the cluster within a few minutes

Expand Down
62 changes: 62 additions & 0 deletions docs/cluster_troubleshoot.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Condo Cluster Deployment Troubleshooting
While Kubernetes combined with the [Git-ops](https://www.weave.works/technologies/gitops/) approach provides a multitude of advantages in terms of scalability and maintainability, can be complicated to set up and diagnose. Below are common commands used to gather more information about your deployments.


### [Command Set 1] Pods within your cluster
A [pod](https://kubernetes.io/docs/concepts/workloads/pods/) is an instance of your running application that has been deployed (although it may still be experiencing issues)
```sh
kubectl get pods {pod-name} -n {namespace} #Get pod within a particular namespace with a particular name
kubectl get pods -n {namespace} #Get pods within a particular namespace
kubectl get pods --all-namespaces #Get all pods within the entire cluster

kubectl describe {pod-name} -n {namespace} #Get more details of a particular pod

kubectl logs {pod-name} #Get log dump of a particular pod
```

### [Command Set 2] Deployments within your cluster
A [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) defines your desired state (configuration) of your application in the cluster.
```sh
kubectl get deployments {deployment-name} -n {namespace} #Get pod within a particular namespace with a particular name
kubectl get deployments -n {namespace} #Get deployments within a particular namespace
kubectl get deployments --all-namespaces #Get all deployments within the entire cluster

kubectl describe {deployment-name} -n {namespace} #Get more details of a particular deployment
```

## Common Issues
### Where is my pod?
After performing a git add and commit to your deployment yaml file within the deploy folder (`UserRoot/.am/clusters/{your-cluster-name}/deploy/`), it usually takes a few minutes for flux to detect and role out the change. The easiest way to check the status of your pod(s) is by finding the pod by the namespace it was supposed to be in (see command 1 above).

---

### I see my pod(s) but they are not ready/show an error
Pods do take time to build, however, they can fail to start or crash during runtime due to a number of reasons. To get more details as to the state of the pod(s) use the `describe` command (see command 1 above). For more in-depth information, consider getting the logs for that pod (see command 1 above).

---

### I have waited 10 minutes, I still do not see my pod
Check to see if flux created a deployment for your application (see command 1 above).

If the deployment exists but pods are not being created, try scaling the pods manually with the following command:
```sh
kubectl scale deployment {deployment-name} -n {namespace} --replicas=2
```
A deployment may have been created but pods were not automatically scaled due to default helm configurations.

---

### I do not see the deployment either
If the deployment does not exist, investigate the flux logs to determine why flux did not create the deployment by using the following commands:

```sh
kubectl get pods -n weave #List all of the pods in the weave namespace

#Identify the flux pod, usually a pod like: flux-7785dfc54d-cx9w2

kubectl logs {flux pod} -n weave #Get logs from flux

#(OPTIONAL)
kubectl logs flux-7785dfc54d-cx9w2 -n weave > flux-logs.txt #pipe flux logs to a text file
```

17 changes: 11 additions & 6 deletions internal/git/gitService.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,12 +91,17 @@ func setUpLocalGitFolder(folderName string, clusterRootPath string, clusterName

func moveCloneIntoLocalRepo(folderName string, clusterRootPath string) {

commandRmGit := exec.Command("rm", "-rf", ".git")
commandRmGit.Dir = clusterRootPath + FPS + "tmp" + FPS + folderName
errRmGit := commandRmGit.Run()
if errRmGit != nil {
log.Fatalf("Error removing git folder at /tmp/"+folderName+". %s", errRmGit)
}
// commandRmGit := exec.Command("rm", "-rf", ".git")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove commented code

// commandRmGit.Dir = clusterRootPath + FPS + "tmp" + FPS + folderName
// errRmGit := commandRmGit.Run()
// if errRmGit != nil {
// log.Fatalf("Error removing git folder at /tmp/"+folderName+". %s", errRmGit)
// }

removeGitError := os.RemoveAll(clusterRootPath + FPS + "tmp" + FPS + folderName + FPS + ".git")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a native way in go to deal with os specific file path creation. filepath.Join()

I have refactored all of the FPS variable usage out of the code in a PR I wrote a while ago but wasn't merged yet. still a little WIP. #18

if removeGitError != nil {
log.Fatal(removeGitError)
}

errMove := os.Rename(clusterRootPath+FPS+"tmp"+FPS+folderName, clusterRootPath+FPS+folderName)

Expand Down