note_time_series.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some important notes on time-series analysis \n",
    "\n",
    "\n",
    "### Nice resources\n",
    "* https://towardsdatascience.com/basic-time-series-manipulation-with-pandas-4432afee64ea\n",
    "* https://medium.com/@bingobee01/pandas-tricks-and-tips-a7b87c3748ea\n",
    "\n",
    "\n",
    "### Obtain the date column\n",
    "Sometimes we have a data set with separate columns of year, month and day here is how we merge them to a single **Date** column.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a single column\n",
    "df['date'] = pd.to_datetime(df[['year', 'month', 'day']].astype(str).agg('-'.join, axis=1))\n",
    "df = df.drop(['year', 'mo', 'da'], axis=1).sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case where you already have a yyyy-mm-dd format, you have to convert it to a panda's datetime object\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Date'] = pd.to_datetime(df['Date'])\n",
    "#convert the date column to index\n",
    "\n",
    "df = df.set_index('Date')\n",
    "\n",
    "# This can also be done while reading the csv file, \n",
    "df = pd.read_csv(filename, parse_dates=True, index_col='Date')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check for missing / duplicate dates\n",
    "The following code requires you to have an index as a datetime object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.date_range(df.index.min(), df.index.max()).difference(df.index)  \n",
    "\n",
    "df = df.reset_index().drop_duplicates(subset='Date', keep='first').set_index('Date')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filling for missing values\n",
    "Use **ffill()** function to fill the missing values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.ffill(axis = 0)  # axis: 0: along the index axis, 1:  along the column axis.\n",
    "\n",
    "df = df.set_index('Date') # first set the Date column as index.\n",
    "\n",
    "df = df.reindex(  pd.date_range(df2.index.min(), df2.index.max())  )  # include the missing dates with NaNs\n",
    "df = df.ffill(axis=0) # fill in the NaNs by using ffill method . Fill a given date by nearest last available point.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train-test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df.loc[ df.Date < datetime(YYYY, MM, DD)  ].copy()\n",
    "df_test = df.loc[ df.Date >= datetime(YYYY, MM, DD)  ].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Merge two data frames together\n",
    "\n",
    "For example if you have two data frames and want to merge them together with the same index set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.concat([df1, df2], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Playing with Date time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TO convert the Date time object in a nice format\n",
    "dt.strftime('%Y-%m-%d')\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}