Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for sentry tracing #517

Merged
merged 4 commits into from
Sep 19, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions ansible/_set_up_directories.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,15 @@
group: deploy
become: true

- name: Move .env file to shared directory
ansible.builtin.template:
src: "templates/dotenv.j2"
dest: "{{ project_root }}/shared/.env"
mode: "0755"
owner: deploy
group: deploy
become: true

- name: Link .env to shared directory
ansible.builtin.file:
src: "{{ project_root }}/shared/.env"
Expand Down
14 changes: 14 additions & 0 deletions ansible/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@
# is a workaround
update_front_end_deps: true

# create new dotenv file in shared directory (use when adding new env vars)
update_dotenv: true

supervisor_restart: true
supervisor_user: "deploy"

Expand All @@ -53,6 +56,17 @@
ansible.builtin.include_tasks: "_assemble_deploy_assets.yml"
when: compile_assets is true

- name: Template .env file to shared directory
ansible.builtin.template:
src: "templates/dotenv.j2"
dest: "{{ project_root }}/shared/.env"
mode: "0755"
owner: deploy
group: deploy
become: true
when: update_dotenv is true
tags: [dotenv]

- name: Set up process management with supervisor
ansible.builtin.include_tasks: "_set_up_process_mgmt.yml"
tags:
Expand Down
5 changes: 5 additions & 0 deletions ansible/inventories/prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,24 @@ all:
internal_ip: "10.0.0.5"
dramatiq_threads: 2
dramatiq_processes: 3
sentry_sample_rate: 0
app2.thegreenwebfoundation.org:
internal_ip: "10.0.0.4"
dramatiq_threads: 2
dramatiq_processes: 3
sentry_sample_rate: 0
# TODO: once we have update the worker process names we can use app3 for serving production traffic
# letting us decommission some of the older app servers
app3.thegreenwebfoundation.org:
internal_ip: "10.0.0.6"
dramatiq_threads: 1
dramatiq_processes: 1
sentry_sample_rate: 1
app4.thegreenwebfoundation.org:
internal_ip: "10.0.0.7"
dramatiq_threads: 2
dramatiq_processes: 3
sentry_sample_rate: 0

vars:
tgwf_stage: "prod"
Expand All @@ -29,6 +33,7 @@ all:
supervisor_user: "deploy"
supervisor_gunicorn_app: "web_{{ tgwf_stage }}"
supervisor_worker_job: "worker_{{ tgwf_stage }}"
gunicorn_port: 9000

# you can set child groups too
children:
Expand Down
3 changes: 2 additions & 1 deletion ansible/inventories/staging.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@ all:
hosts:
app3.thegreenwebfoundation.org:
internal_ip: "10.0.0.6"
gunicorn_port: 10000
dramatiq_threads: 1
dramatiq_processes: 1

sentry_trace_rate: 1
vars:
tgwf_stage: "staging"
tgwf_domain_name: "admin-staging"
Expand Down
5 changes: 5 additions & 0 deletions ansible/templates/dotenv.j2
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,13 @@ EXPLORER_TOKEN="{{ lookup('env','EXPLORER_TOKEN') }}"
SECRET_KEY="{{ lookup('env','SECRET_KEY') }}"
DJANGO_SETTINGS_MODULE="{{ lookup('env','DJANGO_SETTINGS_MODULE') }}"
RABBITMQ_URL="{{ lookup('env', "RABBITMQ_URL") }}"


SENTRY_DSN="{{ lookup('env', "SENTRY_DSN") }}"
SENTRY_ENVIRONMENT="{{ lookup('env', "SENTRY_ENVIRONMENT") }}"
{% if sentry_sample_rate is defined %}
SENTRY_SAMPLE_RATE = {{ sentry_sample_rate }}
{% endif %}

DOMAIN_SNAPSHOT_BUCKET = "{{ lookup('env', 'DOMAIN_SNAPSHOT_BUCKET') }}"
OBJECT_STORAGE_ENDPOINT = "{{ lookup('env', 'OBJECT_STORAGE_ENDPOINT') }}"
Expand Down
2 changes: 1 addition & 1 deletion ansible/templates/run_gunicorn.sh.j2
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# supervisor can only control processes it started itself.
# So we need to use exec to replace the parent shell script process
# that starts pipenv
exec python -m pipenv run gunicorn greenweb.wsgi -b {{ internal_ip }}:$PORT -t 300 -c gunicorn.conf.py --statsd-host=10.0.0.2:9125 --statsd-prefix=member.app
exec python -m pipenv run gunicorn greenweb.wsgi -b {{ internal_ip }}:{{ gunicorn_port }} -t 300 -c gunicorn.conf.py --statsd-host=10.0.0.2:9125 --statsd-prefix=member.app
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We set the port explicitly because this allows us to run production and staging apps on the same machine, without port clashes

4 changes: 0 additions & 4 deletions ansible/templates/supervisor.gunicorn.conf.j2
Original file line number Diff line number Diff line change
@@ -1,19 +1,15 @@
# {{ ansible_managed }}
# Last run: {{ template_run_date }}

[supervisord]
environment=LC_ALL='en_US.UTF-8',LANG='en_US.UTF-8'

[program:{{ supervisor_gunicorn_app }}]
directory=/var/www/{{ tgwf_domain_name }}.thegreenwebfoundation.org/current/
numprocs=1
command=bash ./run_gunicorn.sh
process_name=%(process_num)02d
environment=PORT=90%(process_num)02d
Copy link
Member Author

@mrchrisadams mrchrisadams Sep 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't scale with supervisord - gunicorn has good support for forking subprocesses and threads, and using those to control concurrency, and resource usage, so we do it there instead.

autostart=true
autorestart=true
stopsignal=QUIT
user={{ supervisor_user }}

stdout_logfile=%(program_name)s_%(process_num)02d_.log
stderr_logfile=%(program_name)s_%(process_num)02d_.error.log
2 changes: 1 addition & 1 deletion greenweb/settings/production.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
# set our identifying credentials
dsn=sentry_dsn,
# Set traces_sample_rate.
traces_sample_rate=sentry_sample_rate,
traces_sample_rate=float(sentry_sample_rate),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sentry seems to expect a float, not a string, so we coerce accordingly.

# activate the django specific integrations for sentry
integrations=[DjangoIntegration()],
# We assume that is a user is logged in, we want to be able
Expand Down
Loading