Skip to content

Commit

Permalink
Initial code & example (#1)
Browse files Browse the repository at this point in the history
  • Loading branch information
heronimus authored Sep 25, 2020
1 parent d882408 commit cc0a37a
Show file tree
Hide file tree
Showing 11 changed files with 478 additions and 2 deletions.
235 changes: 233 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,233 @@
# ct-plugin-strings-to-files
Consul-Template plugin to write multiple files from single template.
# Consul-Template Plugin: string2files
Consul-Template plugin that (basically) write string to file(s). Created because I need easy way to write multiple files from Consul KV tree :v:.

### Credits
This plugin heavily inspired by:
- [tam7t/certdump.go Gist](https://gist.github.com/tam7t/1b45125ae4de13b3fc6fd0455954c08e)
- [ekristen/consul-template-plugin-savetofile](https://github.com/ekristen/consul-template-plugin-savetofile)

Both of above code already working nice, but I do some modification and mainly an experiment to try new language ([V-lang](https://github.com/vlang/v)) and make it more lightweight binary (320~ kB) (it's seriously promising new prog. language and fun to try, they have an active community too on Discord).

### What It Can Do
- Write/append string to file.
- Split string and write them to multiple files.
- Allow you to write **multiple files from Consul KV Tree** with only single Consul-Template file.
- Combine Consul KV + Vault Secret and write it to files, and basicly write to file any string you can get from Consul Templates.
- ...

### Installation

- Download the executable binary from [releases page](https://github.com/heronimus/ct-plugin-string2files/releases).
- Give execute permission & copy to your path
```
chmod +x string2files
mv string2files /usr/local/bin/string2files
```

### Usages

Usage: `strings2files [commands] [flags] <arguments..>`

```
Flags:
-help Prints help information.
-f --force Create new directory/file from <path-file> if not exist.
-nl --new-line Add new line in the end of file.
Commands:
append MODE: Append string to file.
create MODE: Write string to file.
explode MODE: Split text and write to multiple file.
help Prints help information.
version Prints version information.
Arguments:
append <file-path> <content>
create <file-path> <content>
explode <base-path> <separator> <content>
```

### Examples (execute binary directly):

- Create file `/opt/files/key-example.txt` with file-content `value-example`:
```
strings2files create /opt/files/key-example.txt "value-example"
```
force directory creation if not exist (`-f`), and add new-line (`-nl`) on the end of the file.
```
strings2files create -f -nl /opt/files/key-example.txt "value-example"
```
<br>
- Append file-content `value-example` to`/opt/files/key-example.txt`:
```
strings2files append /opt/files/key-example.txt "value-example-appended"
```
<br>
- Explode combined key-value strings separated by delimiter:
```
strings2files explode /opt/files/ ";" "key1.txt;value1;key2.txt;value2;child/key3;value3"
```
will create following files tree:
```
/opt/files/
│ key1.txt value1
│ key2.txt value2
└─── child/
│ key3.txt value3
```
<br>
### Examples (as Consul Template Plugin):
These are just a few examples that bring me to create this plugin. Basically, this plugin only writes value given into a file, so it may have many possibilities how you can make use of this plugin.
<br>
For Example: given below items on Consul KV & Vault Secret:
- Consul KV:
```
config/myservices
│ app.conf app.name="myservices"
└─── db/
│ db.conf db.host="127.0.0.1:5432"
```
- Vault Secret:
```
secret/myservices
│ app.conf
| - app.token "s3cr3t-t0k3n"
└─── db/db.conf
│ - db.username "mydb"
│ - db.password "pass1234"
```
<br>
Use cases example:
- **Example 1** (Consul KV Tree): `example/example-consul-kv-tree.tpl`
```
{{ range tree "config/myservices" }}
{{- .Value | plugin "/usr/local/bin/string2files" "create" "-f" "-nl" (print "/opt/myservices/" .Key) -}}
{{ end -}}
```
execute command:
```
consul-template -template "example-consul-kv-tree.tpl:debug.log" -once
```
result:
```
/opt/myservices/
│ app.conf app.name="myservices"
└─── db/
│ db.conf db.host="127.0.0.1:5432"
```
- **Example 2** (Consul KV Tree + Vault Secret appended): `example/example-consul-vault-pair.tpl`
>*Please notes that Consul KV and Vault Secret of this example have same directory structure.
```
{{ range tree "config/myservices" }} {{ $keytemp := .Key }}
{{- .Value | plugin "/usr/local/bin/string2files" "create" "-f" "-nl" (print "/opt/myservices/" .Key) -}}

{{ with secret (print "secret/myservice/" $keytemp) }}{{ range $k, $v := .Data }}
{{ if $k }}
{{- (print $k "=" $v) | plugin "/usr/local/bin/string2files" "append" "-f" "-nl" (print "/opt/myservices/" $keytemp) -}}
{{ end }}{{ end -}}{{ end -}}

{{ end -}}
```
execute command:
```
consul-template -vault-addr <vault-host> -vault-token=<vault-token> -vault-renew-token=false -template "example-consul-vault-pair.tpl:debug.log" -once
```
result:
```
/opt/myservices/
│ app.conf app.name="myservices"
| app.token = s3cr3t-t0k3n
└─── db/
│ db.conf db.host="127.0.0.1:5432"
| db.password = pass1234
| db.username = mydb
```
- **Example 3** (Consul KV Tree with explode mode): `example/example-splits-from-string.tpl`
> *The explode mode allows you to freely construct the data first, then pass the whole strings to the plugin, as the plugin will split the key/value by the separator and create file(s) based on that.
```
{{/* "Initiate Variable" */}}
{{ $separator := "=====" -}}
{{ $data := "" -}}

{{/* "Constructing string to .Data variable" */}}
{{ range tree "config/myservices" }}
{{- $data = (print $data .Key $separator .Value $separator) -}}
{{ end -}}

{{/* "Executing plugin exclude ..." */}}
{{ with $data }}
{{ . | plugin "/usr/local/bin/string2files" "explode" "-f" "-nl" "/opt/myservices/" $separator }}{{ end }}
```
execute command:
```
consul-template -template "example-splits-from-string.tpl:debug.log" -once
```
result:
```
/opt/myservices/
│ app.conf app.name="myservices"
└─── db/
│ db.conf db.host="127.0.0.1:5432"
```
### References
More references about Consul Template and Consul Template plugin, please see official documentation:
- https://github.com/hashicorp/consul-template/#usage
- https://github.com/hashicorp/consul-template/#plugins
### Build from source
This plugin written in V, so you must have V compiler installed.
- Install V (Linux, macOS, Windows, *BSD, Solaris, WSL, Android, Raspbian)
```
git clone https://github.com/vlang/v
cd v
make
```
- Symlink V
- On Unix system
```
sudo ./v symlink
```
- On Windows
```
.\v.exe symlink
```
- Clone this repository
```
git clone https://github.com/heronimus/ct-plugin-string2files.git
```
- Build the code
```
cd ct-plugin-string2files
v string2files.v
```
- Binary will created at working directory (`string2files` / `string2files.exe` for Windows)
> *Notes: V allows you to cross compilation by passing flag `-os <windows/linux>`, macOS binary only can be compiled on macOS platform.
### More About V
Please visit official website and docs:
- https://vlang.io/
- https://github.com/vlang/v/blob/master/doc/docs.md
4 changes: 4 additions & 0 deletions consul-template-example/example-consul-kv-tree.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{{/* "Loop on Consul KV path, then write K/V to files" */}}
{{ range tree "config/myservices" }}
{{- .Value | plugin "/usr/local/bin/string2files" "create" "-f" "-nl" (print "/opt/myservices/" .Key) -}}
{{ end -}}
11 changes: 11 additions & 0 deletions consul-template-example/example-consul-vault-pair.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{{/* "Loop on Consul KV path" */}}
{{ range tree "config/myservices" }} {{ $keytemp := .Key }}
{{- .Value | plugin "/usr/local/bin/string2files" "create" "-f" "-nl" (print "/opt/myservices/" .Key) -}}

{{/* "Get Secret from Vault" */}}
{{ with secret (print "secret/myservice/" $keytemp) }}{{ range $k, $v := .Data }}
{{ if $k }}
{{- (print $k " = " $v) | plugin "/usr/local/bin/string2files" "append" "-f" "-nl" (print "/opt/myservices/" $keytemp) -}}
{{ end }}{{ end -}}{{ end -}}

{{ end -}}
12 changes: 12 additions & 0 deletions consul-template-example/example-splits-from-string.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{{/* "Initiate Variable" */}}
{{ $separator := "=====" -}}
{{ $data := "" -}}

{{/* "Constructing string on $data variable" */}}
{{ range tree "config/myservices" }}
{{- $data = (print $data .Key $separator .Value $separator) -}}
{{ end -}}

{{/* "Executing plugin exclude ..." */}}
{{ with $data }}
{{ . | plugin "/usr/local/bin/string2files" "explode" "-f" "-nl" "/opt/myservices/" $separator }}{{ end }}
17 changes: 17 additions & 0 deletions filewriter/filewriter.v
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
module filewriter

import os
import log

pub struct FileWriter {
mut:
ofile os.File
logger log.Log = log.Log{
level: .info
}
pub mut:
path string
content string
is_force bool
is_newline bool
}
21 changes: 21 additions & 0 deletions filewriter/filewriter_append.v
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
module filewriter

import os

pub fn (mut fw FileWriter) append_file() {
// check force-create directory and file
if !os.is_file(fw.path) {
fw.create_file()
fw.logger.info("file not exist, creating '$fw.path'.")
return
}
// open & append file
fw.ofile = os.open_append(fw.path) or {
fw.logger.error('error while opening file $fw.path for appending')
exit(1)
}
defer {
fw.ofile.close()
}
fw.write_ofile()
}
19 changes: 19 additions & 0 deletions filewriter/filewriter_create.v
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
module filewriter

import os

pub fn (mut fw FileWriter) create_file() {
// check force-create directory
if fw.is_force {
fw.create_dir()
}
// create & write file
fw.ofile = os.create(fw.path) or {
fw.logger.error('error while create file: $err')
exit(1)
}
defer {
fw.ofile.close()
}
fw.write_ofile()
}
29 changes: 29 additions & 0 deletions filewriter/filewriter_explode.v
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
module filewriter

pub fn explode_data(args map[string]string, flags map[string]bool) {
mut fw := FileWriter{
is_force: flags['force']
is_newline: flags['newline']
}
// Split data by separator (bugs on multi char separator)
mut content_split := args['content'].split(args['separator'])
if content_split.len == 0 {
fw.logger.error('KV not found after spliting with separator.')
exit(1)
}
if content_split.len % 2 != 0 {
fw.logger.warn('($content_split) k/v pair is not even, ommiting uncomplete k/v.')
content_split = content_split[0..(content_split.len - 1)]
}
// Loops splited data
for i := 0; i < content_split.len; i += 2 {
if content_split[i] == '' {
fw.logger.warn('path is empty, skipping')
continue
}
fw.logger.info('path --> ${content_split[i]}')
fw.path = args['basepath'] + '/' + content_split[i]
fw.content = content_split[i + 1]
fw.create_file()
}
}
22 changes: 22 additions & 0 deletions filewriter/filewriter_util.v
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
module filewriter

import os

fn (mut fw FileWriter) write_ofile() {
if fw.is_newline {
fw.ofile.writeln(fw.content)
return
}
fw.ofile.write(fw.content)
}

fn (mut fw FileWriter) create_dir() {
dir := os.dir(fw.path)
if !os.is_dir(dir) {
fw.logger.info("directory not exist, creating '$dir'.")
os.mkdir_all(dir)
if !os.is_dir(dir) {
fw.logger.fatal("directory not created: '$dir'.")
}
}
}
Binary file added string2files
Binary file not shown.
Loading

0 comments on commit cc0a37a

Please sign in to comment.