Skip to content

Latest commit

 

History

History
258 lines (178 loc) · 17.5 KB

documentation.md

File metadata and controls

258 lines (178 loc) · 17.5 KB
Raw

Documentation Guidelines

These are the general guidelines I follow for documenting and managing my experiments.

Project management with Asana

I manage all of my research and academic related projects via Asana. Asana is a web-based project management application that allows teams to work together toward accomplishing projects. This allows me to monitor the status of projects and interact with research assistants and our lab manager by assigning tasks or having conversations. For more information on what Asana is, check out these links:

Why we use Asana

From the Asana website:

Between status meetings, scheduling conflicts and never-ending email chains, a lot of your time at work isn’t focused on work. Asana lets you know who is doing what, by when. Communication is focused and actionable and you’ll always know the status of work happening across your team. Spend less time talking about work, and more time actually getting it done.

These points are especially relevant for research projects. Before Asana, I found myself searching through emails or random notes to try to remember what we decided to do and why. Now, all of our conversations about research projects are saved and timestamped. I know exactly when I asked research assistants to do things and why, and they can communicate with me right on a task or project if they have questions or comments.

Before Asana, I found it difficult to keep track of all of the projects I had going on. I spent a lot of hours asking for or generating status updates about projects. Now, I know exactly what is happening and what the status is with all of my projects all the time.

Importantly, when an experiment is complete, I can export the notes and tasks associated with a project as a .pdf file to save with that experiments auxiliary documents.

How we use Asana

Asana allows you to organize tasks into boards (kind of like Trello) or as lists like any standard to-do list application. Further you can cross list a task so that it appears in more than one project. We make use of the boards feature for a high level overview of currently running experiments and writing projects. Each experiment or publication is then cross-listed to an Index page for a collection of experiments so that we can see all of the experiments that have been done within a given genre, and all of the publications that have been written in that genre.

As mentioned above, Asana also allows for chat functionality (kind of like Slack). You can chat with team members about a specific task or start a conversation about a more general project. All chats are searchable and are stored right with the task or project they were initiated in. Anytime someone chats, I am also notified by email and can respond right from my email. Email responses are also saved in the Asana chat.

The most important components of our current Asana workspace are:

Experiments board

The experiments board is a high level overview of all of the experiments we are currently running. Experiments are organized into one of four columns depending on their current status: Waiting on..., In Design, In Data Collection, and Complete.

Each experiment is a task (shown on the board as a card) that corresponds to the naming convention for that experiment: EXPID-collection-descriptive-experiment-title. The task gets assigned to the research assistant or colleague who is doing the subject running and may have a due date that corresponds to when we would like to complete the experiment by. A small chat bubble with a number to the right of it lets us know how many chats are associated with that project.

Experiments are triaged in "Waiting on..." when we are waiting to initiate them or they are next up in the cue. They move to "In Design" when we are in the design phase - creating design sheets, programming and testing the experiment, etc. And they move to "In Data Collection" when we begin collection data. Experiments may move back to "Waiting on..." if they need revision before they can return to data collection.

When an experiment is complete, it is moved to the "Complete" column. These experiments need to be archived, and the experiment task itself does not get checked completed until the checklist for archiving completed experiments has been performed.

Experiment task and sub-task conventions

By clicking on the card, you open the experiment and can view all of its chats and sub-tasks. Each experiment has description and a standard set of task sections:

  1. Background
  2. Designing the experiment
  3. Running the experiment
  4. Transcribing and coding the experiment
  5. Analyzing the data

As of February, 2017, we also try to keep all comments on the main experiment task (and not on sub-tasks), because this allows for easier printing and archiving.

Writing & publishing board

The writing & publishing board is a high level overview of all of the projects we are currently writing up. Writing projects (usually conference abstracts, grants, or papers) are organized into one of five columns depending on their current status: Upcoming abstracts..., Drafting, Waiting for collaborators, Submitted & under review, and In press & published.

The "Upcoming abstracts..." section provides a way for me to view the conferences and grants that I may want to submit to. The "Drafting" section is reserved for papers that I am currently drafting (or will be drafting once data collection is complete). Writing projects move to "Waiting for collaborators" when I have done my part and am now waiting for collaborator comments. They move to "Submitted & under review" when I have submitted them or they are in review.

Finally, when they are in press they move to "In press & published". Writing projects are not checked complete until they are published (or printed for posters, or I have received an acceptance or rejection for grants).

Indexing experiments

Each collection of experiments has an index page that contains cross-listed experiments and writing projects. At the top of each index are the "Publications & Presentations" associated with that collection. This is followed by sub-collections of cross-listed experiments.

For example, my "Empirical Yang" collection contains all of the experiments we have run on our Tolerance Principle work (formation of productive rules). All of the publications and presentations are listed as tasks at the top (these also appear as cards in the "Writing & Publishing" board). Each sub-collection of experiments has a heading (e.g. "Empirical Yang Adult 36-Noun") and each experiment that belongs to that sub-collection is listed below this heading.

File organization system

My research folder is organized by file type. Over the years I have found this to be a more intuitive way for me to find things. Looking for an experiment's design sheet? It is in the design-sheets folder. Looking for the raw data for an experiment? It will be in the raw-data folder. This also drastically simplifies the folder structure. There is a single document or single sub-folder for each experiment within each of the folders shown below. There are no other files stored in any other place.

research-folder-contents

Naming conventions

Within each of these folders, there is a single file or sub-folder for each individual experiment. All files and folders begin with the experiment id, the collection that experiment belongs to, followed by a descriptive title in all lower-case with - as the separator: EXPID-collection-descriptive-title.

This ensure that (1) when sorted alphabetically, the files will arrange by ID number (2) that whenever a file is opened or shared, the filename makes it absolutely clear what it is you are looking it (e.g. there are not a million file folders named "data"), (3) searching by EXPID allows you to find all files related to a given experiment.

Collections of experiments can be identified by the first name in the experiment title, and sub-collections can be identified by the experiment id number (EXPID). You can see how nicely this naming convention works by viewing my lab notebook.

  • subject-tracking
  • stimuli
  • raw-data
  • processed-data
  • experiments
  • docs
  • design-sheets
  • analyses
subject-tracking
  • contains a single .csv with the EXPID-collection-descriptive-title appended with -track.csv
    • For example: 0160-empiricalyang-9noun-hfrule-child-track.csv
stimuli
  • contains a single folder for each experiment with the EXPID-collection-descriptive-title appended with -stims
    • For example: 0160-empiricalyang-9noun-hfrule-child-stims/
  • inside contains all the stimuli for the experiment arranged in sub-folders labeled with the file type (e.g. conditions, sounds, images, movies)
  • may also contain a Word document instructions-text.docx which are the instructions provided to the participant during the experiment.
raw-data
  • contains a single folder for each experiment with the EXPID-collection-descriptive-title appended with -raw-data
    • For example: 0160-empiricalyang-9noun-hfrule-child-raw-data/
  • inside contains all of the raw data for the experiment, which may be single file or folder per participant.
  • subject data folders are named: SID-YEAR-MONTH-DAY-HHMMSS
    • For example: C01-2016-06-17-100307
  • subject data files are named: SID-YEAR-MONTH-DAY-HHMMSS-datatype.csv
    • For example: C01-2016-06-17-100307-exposure.csv
processed-data
  • contains a single folder for each experiment with the EXPID-collection-descriptive-title appended with -processed-data
    • For example: 0160-empiricalyang-9noun-hfrule-child-processed/
  • inside contains all of the processed data for the experiment, which may be
    • an empty text file, indicating that no processed data exists.
    • one or more files with data that has been manipulated in some way
      • preprocessed or transcribed and coded, for example.
experiments
  • contains a single folder for each experiment with the EXPID-collection-descriptive-title appended with -exp
    • For example: 0160-empiricalyang-9noun-hfrule-child-exp/
  • inside is all of the experiment code and files including:
    • change-log.txt: a record of small changes made to the experiment
    • stimuli folders where necessary (e.g. images, conditions)
    • at least one file called: run-exp-v1.py
    • possibly a yaml parameter file
  • Note: While I manage code for creating full experiment paradigms on github, I find this to be overkill for every single experiment. Often an experiment uses a paradigm with no changes except to the stimuli and conditions files, or with only very minor changes. Once a paradigm is working well, I pull the current github version to use in a single experiment. Any modifications to the paradigm that I make for the single experiment get tracked locally in the change-log.txt file. When these small changes are made, I create another version of run-exp in the folder (e.g. run-exp-v2.py). There is a place on the subject tracking sheet to mark which version a participant was run in.
docs
  • contains a single folder for each experiment with the EXPID-collection-descriptive-title appended with -docs
    • For example: 0160-empiricalyang-9noun-hfrule-child-docs/
  • includes any auxilliary documents that do not belong anywhere in the remaining folder architecture but I want to keep.
  • Often these are things like:
    • Data sheet templates for when we record things by hand
    • The call for participants / recruitment procedures
    • Asana notes from the experiment
design-sheets
  • contains a single excel workbook for each experiment EXPID-collection-descriptive-title appended with -design.xslx
    • For example: 0160-empiricalyang-9noun-hfrule-child-design.xslx
  • This is important so that there are not several different excel files and notes containing the experiment design. A single file with a worksheet for each aspect of the design is more than sufficient.
analyses

Location of currently running experiments

Currently running experiments are located in the kschuler-current dropbox folder. This folder is accessible to the Lab's dropbox account (learninglab).

Location of archived experiments

An archive of all experiments is located in the kschuler-all dropbox folder. This folder is accessible to only Katie.

Location of files shared with colleagues

Lab Notebook

As of February 24, 2017 my lab notebook was moved to http://lab.kathrynschuler.com. It is password protected, so you need to request permission from me to view it. As before, it contains a summary of all of the experiments that I have run in graduate school.

Also as of February 24, 2017 I added daily notes on the things I am working on for my future self. This are informal and just intended as a productivity tool for me.

Checklists

Initiating a new experiment

Whenever I begin a new experiment, I follow these steps to ensure the experiment is well documented and organized.

  1. Select a study ID and descriptive title
  2. Add the experiment to the experiment index
  3. Create required files in kschuler-current and populate the folders where needed
    • analyses folder
      • R markdown file
    • design sheet
    • docs folder
      • call for subjects
      • data sheet
    • experiment folder
    • processed-data folder
    • raw-data folder
    • stimuli folder
    • subject-tracking sheet
  4. Add the experiment to the Asana Experiment Board
    • using the new experiment template
  5. Modify the task list of the experiment on Asana
  6. Add the study running information to the guide

Archiving a complete experiment

When an experiment is complete, I move it from the kschuler-current folder (which houses current experiments) to the kschuler-all folder (which houses my experiment archives). To ensure an experiment has been properly archived, I follow these steps:

  1. Open the all-experiments-database.xlsx file.
  2. Fill out all of the Experiment tracking information columns
  3. Move files from kschuler-curent to kschuler-all by following the Documentation checklist on the all-experiments-database.xlsx. For reference, these include:
    • experiment tracking: Have you filled out all of this information?
    • listed in Asana: Is the experiment listed in Asana on the correct index page?
    • analyses: Is the analysis complete and correct?
    • design sheet: Is there a design sheet?
    • docs: Are all docs available? Typically these are:
      • call for participants
      • data sheet
      • notes from asana (printed)
    • experiments: do you have the experiment files? Is the changelog updated?
    • processed data: do you have all of the processed data for this experiment?
    • raw data: do you have all of the raw data for this experiment? including sound files and scanned data sheets?
    • stimuli: have you saved all of the stimuli files for this experiment in this folder?
    • subject tracking: do you have the subject tracking sheet for this experiment?
    • summary: have you saved summary files for this experiment?
      • printed as a .pdf for the folder
      • uploaded to lab notebook server (ghost) for viewing in lab notebook?
    • notes: have you noted anything you will want to remember in the future?

Sending files to colleagues

When a colleague asks me to send them files or other documents, I follow the following steps:

  1. Create a folder for the exchange in my dropbox folder "sent-to-collaborators" in the main dropbox directory.
  2. Name the folder collaborator-date (e.g. jenny-culbertson-2016-11-15) where the date is the date I sent the requested files.
  3. Include all files you want to send inside this folder
  4. Add a text file call materials-requested.txt to this folder that describes what the colleauge needed.
  5. Email the materials.