diff --git a/.gitignore b/.gitignore index 68bc17f9..2de793bd 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,112 @@ +# Created by https://www.toptal.com/developers/gitignore/api/visualstudiocode,pycharm+all,python,venv,jupyternotebooks +# Edit at https://www.toptal.com/developers/gitignore?templates=visualstudiocode,pycharm+all,python,venv,jupyternotebooks +*.csv + +### JupyterNotebooks ### +# gitignore template for Jupyter Notebooks +# website: http://jupyter.org/ + +.ipynb_checkpoints +*/.ipynb_checkpoints/* + +# IPython +profile_default/ +ipython_config.py + +share/ + +# Remove previous ipynb_checkpoints +# git rm -r .ipynb_checkpoints/ + +### PyCharm+all ### +# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider +# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839 + +# User-specific stuff +.idea/**/workspace.xml +.idea/**/tasks.xml +.idea/**/usage.statistics.xml +.idea/**/dictionaries +.idea/**/shelf + +# AWS User-specific +.idea/**/aws.xml + +# Generated files +.idea/**/contentModel.xml + +# Sensitive or high-churn files +.idea/**/dataSources/ +.idea/**/dataSources.ids +.idea/**/dataSources.local.xml +.idea/**/sqlDataSources.xml +.idea/**/dynamic.xml +.idea/**/uiDesigner.xml +.idea/**/dbnavigator.xml + +# Gradle +.idea/**/gradle.xml +.idea/**/libraries + +# Gradle and Maven with auto-import +# When using Gradle or Maven with auto-import, you should exclude module files, +# since they will be recreated, and may cause churn. Uncomment if using +# auto-import. +# .idea/artifacts +# .idea/compiler.xml +# .idea/jarRepositories.xml +# .idea/modules.xml +# .idea/*.iml +# .idea/modules +# *.iml +# *.ipr + +# CMake +cmake-build-*/ + +# Mongo Explorer plugin +.idea/**/mongoSettings.xml + +# File-based project format +*.iws + +# IntelliJ +out/ + +# mpeltonen/sbt-idea plugin +.idea_modules/ + +# JIRA plugin +atlassian-ide-plugin.xml + +# Cursive Clojure plugin +.idea/replstate.xml + +# SonarLint plugin +.idea/sonarlint/ + +# Crashlytics plugin (for Android Studio and IntelliJ) +com_crashlytics_export_strings.xml +crashlytics.properties +crashlytics-build.properties +fabric.properties + +# Editor-based Rest Client +.idea/httpRequests + +# Android studio 3.1+ serialized cache file +.idea/caches/build_file_checksums.ser + +### PyCharm+all Patch ### +# Ignore everything but code style settings and run configurations +# that are supposed to be shared within teams. + +.idea/* + +!.idea/codeStyles +!.idea/runConfigurations + +### Python ### # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] @@ -76,11 +185,8 @@ docs/_build/ target/ # Jupyter Notebook -.ipynb_checkpoints # IPython -profile_default/ -ipython_config.py # pyenv # For a library or package, you might want to ignore these files since the code is @@ -158,3 +264,45 @@ cython_debug/ # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. #.idea/ + +### Python Patch ### +# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration +poetry.toml + +# ruff +.ruff_cache/ + +# LSP config files +pyrightconfig.json + +### venv ### +# Virtualenv +# http://iamzed.com/2009/05/07/a-primer-on-virtualenv/ +[Bb]in +[Ii]nclude +[Ll]ib +[Ll]ib64 +[Ll]ocal +pyvenv.cfg +pip-selfcheck.json + +### VisualStudioCode ### +.vscode/* +!.vscode/settings.json +!.vscode/tasks.json +!.vscode/launch.json +!.vscode/extensions.json +!.vscode/*.code-snippets + +# Local History for Visual Studio Code +.history/ + +# Built Visual Studio Code Extensions +*.vsix + +### VisualStudioCode Patch ### +# Ignore all local history of files +.history +.ionide + +# End of https://www.toptal.com/developers/gitignore/api/visualstudiocode,pycharm+all,python,venv,jupyternotebooks \ No newline at end of file diff --git a/missions/W1/README.md b/missions/W1/README.md new file mode 100644 index 00000000..e2618351 --- /dev/null +++ b/missions/W1/README.md @@ -0,0 +1,101 @@ +# Week1. 환경 설정 + +## 목차 +1. [환경설정](#1-환경설정) + - [pyenv 설치](#pyenv-설치) + - [venv](#venv) + - [jupyter lab, notebook 설치](#jupyter-lab-notebook-설치) + - [jupyter notebook, venv에서 실행하기](#jupyter-notebook-venv에서-실행하기) + - [부록](#부록) + - [참고](#참고) + +## 1. 환경설정 + +### pyenv 설치 +>pyenv는 여러 python 버전을 한 컴퓨터에서 관리 및 사용할 수 있게 해준다. +설치에는 다음의 링크를 참고하자. +[GitHub-pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation) + +어차피 구체적인 환경은 venv를 사용할 예정이므로, pyenv를 통해 설치하는 python은 Global로 설정해도 된다. +```bash +pyenv global 3.12 # python@3.12를 global하게 사용(설치) +``` +
+ +### venv +>venv는 가상환경을 제공해서 워킹디렉토리 별로 python 버전 관리 및 의존성을 따로 관리할 수 있도록 해준다. + +
+ +### jupyter lab, notebook 설치 +[Jupyter Install](https://jupyter.org/install)를 참고하여 주피터 랩과 노트북을 설치한다. +
+ +### jupyter notebook, venv에서 실행하기 +주피터를 이대로 실행하게 되면 Global로 설정된 pyenv를 사용하고 후에 설치하게 될 모든 의존성도 그곳에 모이게된다. +이렇게 될 경우 venv의 이점을 다 누리지 못하므로 우리는 pyenv 대신 venv를 통해 주피터를 사용할 수 있게 설정한다. +
+>pyenv(global)가 아닌 venv(local)에 패키지 설치하고 사용하기 +파이썬이 어디서 실행되는지 알려면 간단하게 which를 사용해볼 수 있다. +``` +which python +``` +![figure-1](assets/figure-1.png) +현재 global로 설정된 pyenv의 python@3.12를 찾아오고 있다. +우리는 이를 venv로 교체한다. +
+W1 폴더에 venv를 만들어준다. +```bash +python -m venv +``` +![figure-2](assets/figure-2.png) +
+가상환경을 활성화한다. +```bash +source /bin/activate +``` +![figure-3](assets/figure-3.png) +
+이제 주피터 노트북과 venv를 연결해줘야 한다. +ipykernel을 설치하자. +- ipykernel도 가상환경 밑에 설치되도록, activate 이후 설치한다. +```bash +pip install ipykernel +``` +
+ +커널을 하나 만들어준다. 추가할 가상환경과 주피터에서 display할 이름을 정의한다. +```bash +python -m ipykernel install --user --name [가상환경폴더] --display-name [Jupyter에서 보여질 이름] +# python -m ipykernel install --user --name W1 --display-name W1-venv +``` + +
+ +주피터 실행 후 ipynb 파일을 생성하고 커널을 변경한다. +![figure-4](assets/figure-4.png) + +
+ +새로운 패키지를 설치해보고 올바른 위치(venv)에 깔리는지 확인하자. +![figure-5](assets/figure-5.png) + +
+ +#### 부록. +현재 존재하는 커널 리스트를 확인하고 삭제하는 방법 +```bash +# List all kernels and grap the name of the kernel you want to remove +jupyter kernelspec list +# Remove it +jupyter kernelspec remove +``` + +
+ +##### 참고. +[JupyterLab 에 가상환경(Virtualenv) 연결, 삭제하기 +](https://raki-1203.github.io/jupyter/JupyterLab_venv_add_delete/) + + + diff --git a/missions/W1/assets/figure-1.png b/missions/W1/assets/figure-1.png new file mode 100644 index 00000000..3aeb06b2 Binary files /dev/null and b/missions/W1/assets/figure-1.png differ diff --git a/missions/W1/assets/figure-2.png b/missions/W1/assets/figure-2.png new file mode 100644 index 00000000..98a26cba Binary files /dev/null and b/missions/W1/assets/figure-2.png differ diff --git a/missions/W1/assets/figure-3.png b/missions/W1/assets/figure-3.png new file mode 100644 index 00000000..85f93d51 Binary files /dev/null and b/missions/W1/assets/figure-3.png differ diff --git a/missions/W1/assets/figure-4.png b/missions/W1/assets/figure-4.png new file mode 100644 index 00000000..586cd102 Binary files /dev/null and b/missions/W1/assets/figure-4.png differ diff --git a/missions/W1/assets/figure-5.png b/missions/W1/assets/figure-5.png new file mode 100644 index 00000000..35b800a7 Binary files /dev/null and b/missions/W1/assets/figure-5.png differ diff --git a/missions/W1/mtcars.csv b/missions/W1/mtcars.csv deleted file mode 100644 index a22b9c21..00000000 --- a/missions/W1/mtcars.csv +++ /dev/null @@ -1,33 +0,0 @@ -"","mpg","cyl","disp","hp","drat","wt","qsec","vs","am","gear","carb" -"Mazda RX4",21,6,160,110,3.9,2.62,16.46,0,1,4,4 -"Mazda RX4 Wag",21,6,160,110,3.9,2.875,17.02,0,1,4,4 -"Datsun 710",22.8,4,108,93,3.85,2.32,18.61,1,1,4,1 -"Hornet 4 Drive",21.4,6,258,110,3.08,3.215,19.44,1,0,3,1 -"Hornet Sportabout",18.7,8,360,175,3.15,3.44,17.02,0,0,3,2 -"Valiant",18.1,6,225,105,2.76,3.46,20.22,1,0,3,1 -"Duster 360",14.3,8,360,245,3.21,3.57,15.84,0,0,3,4 -"Merc 240D",24.4,4,146.7,62,3.69,3.19,20,1,0,4,2 -"Merc 230",22.8,4,140.8,95,3.92,3.15,22.9,1,0,4,2 -"Merc 280",19.2,6,167.6,123,3.92,3.44,18.3,1,0,4,4 -"Merc 280C",17.8,6,167.6,123,3.92,3.44,18.9,1,0,4,4 -"Merc 450SE",16.4,8,275.8,180,3.07,4.07,17.4,0,0,3,3 -"Merc 450SL",17.3,8,275.8,180,3.07,3.73,17.6,0,0,3,3 -"Merc 450SLC",15.2,8,275.8,180,3.07,3.78,18,0,0,3,3 -"Cadillac Fleetwood",10.4,8,472,205,2.93,5.25,17.98,0,0,3,4 -"Lincoln Continental",10.4,8,460,215,3,5.424,17.82,0,0,3,4 -"Chrysler Imperial",14.7,8,440,230,3.23,5.345,17.42,0,0,3,4 -"Fiat 128",32.4,4,78.7,66,4.08,2.2,19.47,1,1,4,1 -"Honda Civic",30.4,4,75.7,52,4.93,1.615,18.52,1,1,4,2 -"Toyota Corolla",33.9,4,71.1,65,4.22,1.835,19.9,1,1,4,1 -"Toyota Corona",21.5,4,120.1,97,3.7,2.465,20.01,1,0,3,1 -"Dodge Challenger",15.5,8,318,150,2.76,3.52,16.87,0,0,3,2 -"AMC Javelin",15.2,8,304,150,3.15,3.435,17.3,0,0,3,2 -"Camaro Z28",13.3,8,350,245,3.73,3.84,15.41,0,0,3,4 -"Pontiac Firebird",19.2,8,400,175,3.08,3.845,17.05,0,0,3,2 -"Fiat X1-9",27.3,4,79,66,4.08,1.935,18.9,1,1,4,1 -"Porsche 914-2",26,4,120.3,91,4.43,2.14,16.7,0,1,5,2 -"Lotus Europa",30.4,4,95.1,113,3.77,1.513,16.9,1,1,5,2 -"Ford Pantera L",15.8,8,351,264,4.22,3.17,14.5,0,1,5,4 -"Ferrari Dino",19.7,6,145,175,3.62,2.77,15.5,0,1,5,6 -"Maserati Bora",15,8,301,335,3.54,3.57,14.6,0,1,5,8 -"Volvo 142E",21.4,4,121,109,4.11,2.78,18.6,1,1,4,2 diff --git a/missions/W3/mapreduce-sentimental/Dockerfile b/missions/W3/mapreduce-sentimental/Dockerfile new file mode 100644 index 00000000..076609e7 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/Dockerfile @@ -0,0 +1,69 @@ +FROM ubuntu:22.04 + +ENV PDSH_RCMD_TYPE=ssh +ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-arm64 +# Set Hadoop ENV +ENV HADOOP_VERSION=3.4.0 +ENV HADOOP_HOME=/opt/hadoop +ENV HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop +ENV PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin +# Set Hadoop User +ENV HDFS_NAMENODE_USER=hdfs +ENV HDFS_DATANODE_USER=hdfs +ENV HDFS_SECONDARYNAMENODE_USER=hdfs +ENV YARN_RESOURCEMANAGER_USER=yarn +ENV YARN_NODEMANAGER_USER=yarn + +RUN apt-get update && \ + apt-get install -y openjdk-11-jdk ssh pdsh wget sudo && \ + apt-get clean + +# Download and extract Hadoop to /opt +RUN wget https://dlcdn.apache.org/hadoop/common/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz -P /opt && \ + tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt && \ + mv /opt/hadoop-$HADOOP_VERSION $HADOOP_HOME && \ + rm /opt/hadoop-$HADOOP_VERSION.tar.gz + +RUN apt-get install -y sshpass vim && \ + sudo apt-get install -y jq && \ + wget https://github.com/mikefarah/yq/releases/download/v4.16.1/yq_linux_amd64 -O /usr/bin/yq && \ + chmod +x /usr/bin/yq + +# 사용자 및 그룹 생성 및 권한 설정 +RUN groupadd -g 1000 hadoop && \ + useradd -m -u 1001 -g 1000 hdfs && \ + useradd -m -u 1002 -g 1000 yarn && \ + echo "hdfs ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers && \ + echo "yarn ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + +USER hdfs +RUN ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa && \ + mkdir -p ~/.ssh && \ + cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && \ + chmod 0600 ~/.ssh/authorized_keys + +USER yarn +RUN ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa && \ + mkdir -p ~/.ssh && \ + cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && \ + chmod 0600 ~/.ssh/authorized_keys + +USER root +RUN ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa && \ + mkdir -p ~/.ssh && \ + cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && \ + chmod 0600 ~/.ssh/authorized_keys + +RUN mkdir -p $HADOOP_HOME/logs && mkdir -p $HADOOP_HOME/dfs && \ + chown -R :hadoop $HADOOP_HOME/ && \ + chmod -R 775 $HADOOP_HOME/ + +RUN echo "export JAVA_HOME="$(jrunscript -e 'java.lang.System.out.println(java.lang.System.getProperty("java.home"));')"" >> $HADOOP_CONF_DIR/hadoop-env.sh + +COPY configs/* $HADOOP_CONF_DIR/ +COPY scripts/* /usr/local/bin/ +RUN chmod +x /usr/local/bin/*.sh + +WORKDIR $HADOOP_HOME + +ENTRYPOINT ["/usr/local/bin/start.sh"] \ No newline at end of file diff --git a/missions/W3/mapreduce-sentimental/README.md b/missions/W3/mapreduce-sentimental/README.md new file mode 100644 index 00000000..5057bd13 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/README.md @@ -0,0 +1,4 @@ +# MapReduce Twitter Sentimental + +아래 링크에 정리했습니다. +https://jang-namu.notion.site/M4-MapReduce-Twitter-Sentimental-ce80b2d072c04ec7b1de8202313dd603?pvs=4 \ No newline at end of file diff --git a/missions/W3/mapreduce-sentimental/configs/core-site.xml b/missions/W3/mapreduce-sentimental/configs/core-site.xml new file mode 100644 index 00000000..1ef4758b --- /dev/null +++ b/missions/W3/mapreduce-sentimental/configs/core-site.xml @@ -0,0 +1,10 @@ + + + fs.defaultFS + hdfs://hadoop-master:9000 + + + hadoop.tmp.dir + /tmp/hadoop-${user.name} + + diff --git a/missions/W3/mapreduce-sentimental/configs/hdfs-site.xml b/missions/W3/mapreduce-sentimental/configs/hdfs-site.xml new file mode 100644 index 00000000..96b9c95e --- /dev/null +++ b/missions/W3/mapreduce-sentimental/configs/hdfs-site.xml @@ -0,0 +1,18 @@ + + + dfs.replication + 2 + + + dfs.namenode.name.dir + file:///opt/hadoop/dfs/namenode + + + dfs.datanode.data.dir + file:///opt/hadoop/dfs/datanode + + + dfs.namenode.checkpoint.dir + file:///opt/hadoop/dfs/namesecondary + + diff --git a/missions/W3/mapreduce-sentimental/configs/mapred-site.xml b/missions/W3/mapreduce-sentimental/configs/mapred-site.xml new file mode 100644 index 00000000..a8a8c030 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/configs/mapred-site.xml @@ -0,0 +1,12 @@ + + + mapreduce.framework.name + yarn + + + mapreduce.application.classpath + $HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/* + + + + diff --git a/missions/W3/mapreduce-sentimental/configs/masters b/missions/W3/mapreduce-sentimental/configs/masters new file mode 100644 index 00000000..22a6c304 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/configs/masters @@ -0,0 +1 @@ +hadoop-master \ No newline at end of file diff --git a/missions/W3/mapreduce-sentimental/configs/workers b/missions/W3/mapreduce-sentimental/configs/workers new file mode 100644 index 00000000..f2fa46c9 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/configs/workers @@ -0,0 +1,2 @@ +hadoop-slave1 +hadoop-slave2 \ No newline at end of file diff --git a/missions/W3/mapreduce-sentimental/configs/yarn-site.xml b/missions/W3/mapreduce-sentimental/configs/yarn-site.xml new file mode 100644 index 00000000..73728207 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/configs/yarn-site.xml @@ -0,0 +1,14 @@ + + + yarn.resourcemanager.hostname + hadoop-master + + + yarn.nodemanager.aux-services + mapreduce_shuffle + + + + + + diff --git a/missions/W3/mapreduce-sentimental/docker-compose.yml b/missions/W3/mapreduce-sentimental/docker-compose.yml new file mode 100644 index 00000000..2d208543 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/docker-compose.yml @@ -0,0 +1,51 @@ +services: + hadoop-master: + image: jnamu/mapreduce-node:latest + container_name: hadoop-master + hostname: hadoop-master + ports: + - "9870:9870" # HDFS NameNode Web UI + - "8020:8020" # HDFS NameNode HDFS API + - "8088:8088" # YARN ResourceManager Web UI + - "9000:9000" # fs.DefaultFS + volumes: + - hdfs-master:/opt/hadoop/dfs + networks: + - hadoop + + hadoop-slave1: + image: jnamu/mapreduce-node:latest + container_name: hadoop-slave1 + hostname: hadoop-slave1 + depends_on: + - hadoop-master + ports: + - "9864:9864" # HDFS DataNode Web UI + - "8042:8042" # YARN NodeManager Web UI + volumes: + - hdfs-slave-1:/opt/hadoop/dfs + networks: + - hadoop + + hadoop-slave2: + image: jnamu/mapreduce-node:latest + container_name: hadoop-slave2 + hostname: hadoop-slave2 + depends_on: + - hadoop-master + ports: + - "9865:9864" # HDFS DataNode Web UI + - "8043:8042" # YARN NodeManager Web UI + volumes: + - hdfs-slave-2:/opt/hadoop/dfs + networks: + - hadoop + +networks: + hadoop: + driver: bridge + +volumes: + hdfs-master: + hdfs-slave-1: + hdfs-slave-2: diff --git a/missions/W3/mapreduce-sentimental/scripts/mapper.py b/missions/W3/mapreduce-sentimental/scripts/mapper.py new file mode 100644 index 00000000..2ad3db29 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/scripts/mapper.py @@ -0,0 +1,12 @@ +#!/usr/bin/env python3 +import sys + +for line in sys.stdin: + values = line.strip().split(",") + + if values[0] == '"0"': + print('negative\t1') + elif values[0] == '"2"': + print('neutral\t1') + else: + print('positive\t1') diff --git a/missions/W3/mapreduce-sentimental/scripts/reducer.py b/missions/W3/mapreduce-sentimental/scripts/reducer.py new file mode 100644 index 00000000..92231394 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/scripts/reducer.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +import sys + +current_sentiment = None +current_count = 0 +sentiment = None + +for line in sys.stdin: + line = line.strip() + sentiment, count = line.split('\t', 1) + + count = int(count) + + if current_sentiment == sentiment: + current_count += count + else: + if current_sentiment: + print(f'{current_sentiment}\t{current_count}') + current_sentiment = sentiment + current_count = count + +if current_sentiment == sentiment: + print(f'{current_sentiment}\t{current_count}') \ No newline at end of file diff --git a/missions/W3/mapreduce-sentimental/scripts/start.sh b/missions/W3/mapreduce-sentimental/scripts/start.sh new file mode 100644 index 00000000..72df051d --- /dev/null +++ b/missions/W3/mapreduce-sentimental/scripts/start.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash + +# Start SSH service +service ssh start + +# Format namenode if not formatted +if [ "$HOSTNAME" = "hadoop-master" ]; then + echo "Checking if namenode format is needed..." + if [ ! -d "$HADOOP_HOME/dfs/namenode/current" ]; then + echo "Formatting namenode..." + sudo -E -u hdfs $HADOOP_HOME/bin/hdfs namenode -format + fi +fi + +# Start Hadoop services +if [ "$HOSTNAME" = "hadoop-master" ]; then + echo "Start hadoop-master node daemon" + sudo -E -u hdfs $HADOOP_HOME/bin/hdfs --daemon start namenode + sudo -E -u hdfs $HADOOP_HOME/bin/hdfs --daemon start secondarynamenode + sudo -E -u yarn $HADOOP_HOME/bin/yarn --daemon start resourcemanager + sudo -E -u yarn $HADOOP_HOME/bin/yarn --daemon start nodemanager + + echo "create directory /user/root & change owner" + sudo -E -u hdfs $HADOOP_HOME/bin/hdfs dfs -mkdir -p /user/root/ + sudo -E -u hdfs $HADOOP_HOME/bin/hdfs dfs -chown root:root /user/root +else + echo "Start hadoop-worker datanode daemon" + sudo -E -u hdfs $HADOOP_HOME/bin/hdfs --daemon start datanode + echo "Start hadoop-worker nodemanager daemon" + sudo -E -u yarn $HADOOP_HOME/bin/yarn --daemon start nodemanager +fi + +# Keep the container running +echo "Keep foreground process to maintain running" +tail -f /dev/null \ No newline at end of file diff --git a/missions/W3/mapreduce-sentimental/volume-recreate.sh b/missions/W3/mapreduce-sentimental/volume-recreate.sh new file mode 100755 index 00000000..a2e349e4 --- /dev/null +++ b/missions/W3/mapreduce-sentimental/volume-recreate.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash + +docker volume rm hdfs-master hdfs-slave-1 hdfs-slave-2 +docker volume create hdfs-master +docker volume create hdfs-slave-1 +docker volume create hdfs-slave-2 + + + +