Develop (#439)

* Fix: charger ids fix * #81 Fork changes in abstract class. * #81 Data packer. * #81 Revert registrations (draft 1). * #81 Revert registrations (draft 2). * #81 Remove revert registration for Genesis. * #81 Register fork changes draft. * #81 Refactor votes. * #81 Register draft. * #81 Subcribes refactoring. * fix: payout calculation, bignum * #81 Leader fixes. * #81 Fix profile. * feat: add remove reblog handler #255 * #81 Fixes. * Fix: battery calculation * fix: disable genesis block handler * fix: update gls-core-service version * #81 #154 #153 #222 Profile. * #81 Update core. * #81 Merge fixes. * #81 Review fixes. * Fix: elasticsearch deploy config * #81 Review fixes 2. * #285 Leader model fixes. * #285 Review fixes. * #288 Revert last block on start. * #288 Review fixes. * #289 Feat: add charger restore formula basement * refacot: changereput to votestate * feat: use new BlockSubscribe #295 * Refactor: divide raw and calculated chargers * Refactor: divide raw and calculated chargers * Refactor: divide raw and calculated chargers * Refactor: divide raw and calculated chargers * Refactor: divide raw and calculated chargers * Fix: unexpected error when fetching profile which does not exist * Votes hotfix. * Votes hotfix 2. * Votes hotfix 3. * Votes hotfix 4. * Votes hotfix 5. * Metadata hotfix. * fix: add metrics endpoint * fix: isSubscribed * Update core. * Update core 2. * Update core 3. * Update core 3. * Update core 4. * Vote state hotfix. * Fork system hotfix. * Wait for transaction hotfix. * Chargers hotfix. * Payout hotfix. * feat: save social networks in contacts for gls * fix: vk to vkontakte * feat: repost time #308 * refactor: improve * #302 Fix revert last block. * feat: fact payouts after post close * refactor: improve * fix: review * fix: review * fix: review * fix: review * Api hotfix. * Votes hotfix + core update. * Remove unused packages. * Npmrc fix. * #313 User reputation in feed. * Connector hotfix. * ContentId query fixes. * feat: implement genesis applying #270 fix: set MaxInFlight to 50 feat: add genesis example saving #270 fix: remove external mongodb mount fix: metrics names feat: add genesis witnessstate processing #270 fix: review fixes #270 fix: new gls-core-service fixes #270 fix: review fixes #270 * fix: shrink nats errors * fix: update ServiceMeta * fix: remove fork cleaning logging * fix: fix post/comments indexes * feat: add metrics * fix: use GenesisProcessor from core #270 * refactor: move BulkSaver to core * #318 Fix tags query. * fix: update gls-core-service version * fix: add post/comments creation time * fix: review fixes * fix: review fixes * Feed indexes hotfix. * Feed indexes hotfix 2. * Async start feed cache. * Feed sync mode on. * Feed cache on stream. * feat: payouts from genesis * Make isRepost false in search. * fix: improve * Log feed cache time. * Log feed cache iteration. * fix: improve * Add pool size. * feat: reblogs from genesis * fix: update witnesssate processing * fix: update lodash version * Modes. * fix: update indexes * fix: update post indexes * fix: try to fix indexes * fix: try to fix indexes * #324 Avatar in comments fix. * fix: new BlockSubscribe api (#333) * #324 Avatar in comments fix (#336) * fix: getPost api method #338 * feat: rShares and Wilson scoring #340 * #337 Restore feed sort validation. * fix: fix feed reposts * feat: fork revert logic #301 * fix: fix leaders ordering (#349) * fix: fix block subscribe logic * proposals: add approves list * feat: process community settings #354 * feat: add executed flag into proposals #354 * fix: review fixes #354 * #348 Repost fix. * fix: review fixes #354 * Broadcaster doc. * feat: add "symbol" to vesting contract settings * Feat: return repost title * Revert "feat: add "symbol" to vesting contract settings" (#363) This reverts commit 3a529f5. * Sort comments byTime hotfix. * Hotfix comments sort. * Feat: return repost title Refactor: apply prettier * Feat: add flag in profile for detecting genesis users * feat: detect vesting opening #366 * fix: add field "email" into gls account personal (#368) * fix: don't search votes in reposts * fix: improve * feat: calculation of payouts, unclaimed payouts (#375) * fix: upVotes is not iterable * fix: improve * fix: negative unclaimed payout #378 * Feat: personal feed * Hotfix: comment votes * Hotfix: fix hotfix * Fix: parse reputation * #382 Feat: get content by any name * Fix: remove unused dep * Fix: naming * fix: fix leaders ordering #384 * fix: improve Leaders indexes #384 * Fixes (#386) * Fix: promise handling * Fix: add index needed for replies * Small fix (#388) * Fix: promise handling * Fix: add index needed for replies * Fix: add user param * Fix: genesis account parsing (#391) * Fix: genesis account parsing * feat: add charge::setrestorer proposal processing #389 (#390) * feat: add charge::setrestorer proposal processing #389 * fix: review fixes * Fix battery formula (#392) * Fix: chargers formula * Styling: apply prettier * fix: fix feed sequence key #387 (#393) * feat: populate repost author in feed (#396) * fix: fill leaderIn from genesis * Feat: add find leader by username method (#397) * Feat: add find leader by username method * Refactor: regexp string * Fix find leaders method (#398) * Feat: add find leader by username method * Refactor: regexp string * Fix: find leaders * fix: fix regex query (#399) * Fix: forks issue (#400) * fix: fix findLeaders method (#401) * fix: fix findLeaders method (2) * fix: fix findLeaders method (3) * Fix leaders (#402) * Refactor: leadersIn mechanics in find leaders method * Refactor: leadersIn mechanics in find leaders method * Refactor: leadersIn mechanics in find leaders method * Fix: query * Fix leaders (#403) * Refactor: leadersIn mechanics in find leaders method * Refactor: leadersIn mechanics in find leaders method * Refactor: leadersIn mechanics in find leaders method * Fix: query * Fix: query * feat: add position field to leaders (and refactor leaders) (#404) * feat: add position field to leaders * fix: review fixes * fix: fix getLeadersTop * fix: fix leaders ordering * Fix: replies index (#405) * Fix: replies index * Fix: replies index * fix: unclaimed calculation in payouts (#408) * fix: make username greater priority over userId #406 * fix: fix AbstractFeed (#410) * fix: fix leaders vote status (#412) * refactor: improve proposal processing (#413) * fix: make docker-compose.yml as example (#414) * fix: if username is not specified return as "null" (#417) * feat: add publish.setrules processing (#418) * fix: fix accounts from genesis (#419) * fix: fix Leaders vote processing (#420) * fix: fix leaders unvote processing (#421) * fix: fix CommunitySettings * feat: add method "getUsernames" (#422) * feat: add custom proposals processing #423 * fix: proposals query * fix: proposals query * fix: use "cyberway-core-service" package (#425) * fix: update cyberway-core-service version * fix: update cyberway-core-service version * fix: update cyberway-core-service verstion * fix: update cyberway-core-service version * feat: add multi node start for production (#426) * fix: reduce unused logs * fix: fix leaders proposals page (#427) * feat: add "setrules" proposals processing (#428) * fix: handle proposal cancelation (#429) * fix: fix getProposals pagination (#430) * fix: fix getProposals pagination * fix: review fixes * fix: fix extract body (#432) * refactor: prepare for change nats (#433) * fix: do not request own comments in replies #434 (#435) * Fix reposts (#436) * Refactor: leadersIn mechanics in find leaders method * Fix: reposts parsing * Refactor: apply prettier * Fix: popular tags sorting: by time now (#437) * fix applyParentById Co-authored-by: IvanBlacky <[email protected]> Co-authored-by: Oleg <[email protected]> Co-authored-by: Alexey Elizarov <[email protected]> Co-authored-by: Ilya Lebedev <[email protected]>
GolosChain · Sep 4, 2020 · 3785f38 · 3785f38
1 parent b7e5f96
commit 3785f38
Show file tree

Hide file tree

Showing 62 changed files with 4,820 additions and 1,764 deletions.
diff --git a/.env.example b/.env.example
@@ -1,13 +1,15 @@
-NODE_OPTIONS=--max_old_space_size=8192
-GLS_DAY_START=3
-GLS_MONGO_CONNECT=mongodb://mongo/admin
-GLS_METRICS_HOST=127.0.0.1
-GLS_METRICS_PORT=8125
-GLS_CONNECTOR_HOST=127.0.0.1
+GLS_PRISM_NODE_OPTIONS=--max_old_space_size=8192
+GLS_CONNECTOR_HOST=0.0.0.0
 GLS_CONNECTOR_PORT=3000
+GLS_METRICS_HOST=0.0.0.0
+GLS_METRICS_PORT=9777
+GLS_CONNECTOR_PRISM_HOST=0.0.0.0
+GLS_CONNECTOR_PRISM_PORT=3001
+GLS_METRICS_PRISM_HOST=0.0.0.0
+GLS_METRICS_PRISM_PORT=9778
+GLS_DAY_START=3
+GLS_MONGO_CONNECT=mongodb://prism-mongo/admin
 GLS_MAX_FEED_LIMIT=100
-GLS_DELEGATION_ROUND_LENGTH=21
-GLS_REVERT_TRACE_CLEANER_INTERVAL=300000
 GLS_BLOCKCHAIN_BROADCASTER_SERVER_NAME=
 GLS_BLOCKCHAIN_BROADCASTER_CLIENT_NAME=
 GLS_BLOCKCHAIN_BROADCASTER_CONNECT=
@@ -29,3 +31,4 @@ GLS_EXPORT_GOLOS_USERS_CONNECT=
 GLS_SEARCH_ENABLED=true
 GLS_USE_GENESIS=true
 GLS_META_CONNECT=
+GLS_MAX_QUERY_MEMORY_LIMIT=536870912
diff --git a/.gitignore b/.gitignore
@@ -4,3 +4,5 @@ db
 .env
 .vscode
 .npmrc
+docker-compose.yml
+stats.txt
diff --git a/Dockerfile b/Dockerfile
@@ -1,10 +1,6 @@
-FROM node:10 as builder
+FROM node:10
 WORKDIR /usr/src/app
 COPY ./package*.json ./
 RUN npm install --only=production
-
-FROM node:10
-WORKDIR /usr/src/app
-COPY --from=builder /usr/src/app .
 COPY ./src/ ./src
 CMD [ "node", "./src/index.js" ]
diff --git a/README.md b/README.md
@@ -3,11 +3,18 @@
 **PRISM-SERVICE** является сервисом дисперсии данных блокчейна CyberWay, которые после вторичной обработки другими
 микросервисами могут быть использованы в [golos.io](https://golos.io) и приложениях.
 
-Запуск:
+### Запуск:
 
 -   Установить `docker` и `docker-compose`
 -   Установить необходимые ENV-переменные в `.env` файл (шаблон есть в `.env.example`)
--   Вызвать команду `docker-compose up --build` в корне проекта
+-   Скопировать и переименовать нужный docker-compose файл в `docker-compose.yml`.
+-   Вызвать команду `docker-compose up -d --build` в корне проекта
+
+Существует два варианта запуска сервиса:
+
+-   одно-нодовый (`docker-compose.dev.yml`) где процесс обработки новых блоков происходит в том же процессе что и выдача данных;
+-   двух-нодовый (`docker-compose.multi-node.yml`) где процессы обработки блоков и обработка API разделены на два независимых процесса, рекомендованно для запуска на продакшене.
+
 
 API JSON-RPC:
 
@@ -40,6 +47,7 @@ search:                            // Поиск по данным из приз
 getProfile:                        // Получение профиля пользователя
     requestedUserId <string>       // Идентификатор пользователя
     username <string>              // Имя пользователя относительно домена
+    user <string>                  // Любое из имен выше
     app <string>('cyber')          // Тип приложения / домена
         [
           cyber                    // CyberWay
@@ -53,10 +61,14 @@ getProfile:                        // Получение профиля поль
         | gls                      // Golos
         ]
 
+getChargers:                       // Получение батареек пользователя
+    userId <string>                // Идентификатор пользователя
+
 getPost:                           // Получение конкретного поста
     currentUserId <string/null>    // Идентификатор текущего пользователя
     requestedUserId <string>       // Идетификатор запрошенного пользователя
     username <string>              // Имя пользователя относительно домена
+    user <string>                  // Любое из имен выше
     app <string>('cyber')          // Тип приложения / домена
         [
           cyber                    // CyberWay
@@ -74,6 +86,7 @@ getComment:                        // Получение конкретного
     currentUserId <string/null>    // Идентификатор текущего пользователя
     requestedUserId <string/null>  // Идетификатор запрошенного пользователя
     username <string>              // Имя пользователя относительно домена
+    user <string>                  // Любое из имен выше
     app <string>('cyber')          // Тип приложения / домена
         [
           cyber                    // CyberWay
@@ -252,6 +265,7 @@ getLeadersTop:                   // Получить топ лидеров
     communityId <string>         // Идентификатор комьюнити
     sequenceKey <string/null>    // Идентификатор пагинации для получения следующего контента
     limit <number>(10)           // Количество элементов
+    query <string>               // Префиксный поиск по имени аккаунта
     app <string>('cyber')        // Тип приложения / домена
         [
           cyber                  // CyberWay
@@ -299,12 +313,6 @@ waitForTransaction               // Дождаться и получить от
 -   `GLS_MAX_FEED_LIMIT` - максимальное количество постов отдаваемое в ленту на 1 запрос за 1 раз.  
     Дефолтное значение - `100`
 
--   `GLS_DELEGATION_ROUND_LENGTH` - количество раундов подписи блока в блокчейне.  
-    Дефолтное значение - `21`
-
--   `GLS_REVERT_TRACE_CLEANER_INTERVAL` - интервал запуска клинера неактуальных записей восстановления в случае форков.  
-    Дефолтное значение - `300000` _(5 минут)_
-
 -   `GLS_FEED_CACHE_INTERVAL` - интервал перерассчета кешей для кешируемых типов лент (но полное удаление старого происходит по `GLS_FEED_CACHE_TTL`).  
     Дефолтное значение - `300000` _(5 минут)_
 
@@ -338,5 +346,20 @@ waitForTransaction               // Дождаться и получить от
 -   `GLS_SEARCH_ENABLED` - определяет, должно ли происходить индексирование для полнотекстого поиска.
     Дефолтное значение - `true`
 
--   `GLS_USE_GENESIS` - необходимо ли ожидать загрузки генезис-данных.
+-   `GLS_USE_GENESIS` - необходимо ли ожидать загрузки генезис-данных.  
     Дефолтное значение - `true`
+
+-   `GLS_MAX_QUERY_MEMORY_LIMIT` - лимит памяти для одного запроса, критично для системы кеширования.  
+    Дефолтное значение - `536870912` _(512 Мб)_
+
+-   `GLS_ENABLE_BLOCK_HANDLE` - включает сбор данных из блокчейна, а также апи ожидания транзакций.  
+    Дефолтное значение - `true`
+
+-   `GLS_ENABLE_PUBLIC_API` - включает выдачу данных по публичному апи, за исключением апи ожидания транзакций.  
+    Дефолтное значение - `true`
+
+-   `GLS_BLOCKCHAIN_BROADCASTER_SERVER_NAME` - имя сервера рассыльщика блоков.
+
+-   `GLS_BLOCKCHAIN_BROADCASTER_CLIENT_NAME` - имя клиента для подключения к рассыльщику блоков.
+
+-   `GLS_BLOCKCHAIN_BROADCASTER_CONNECT` - строка подключения к рассыльщику блоков, может содержать авторизацию.
diff --git a/docker-compose.yml → docker-compose.dev.yml b/docker-compose.yml → docker-compose.dev.yml
@@ -23,22 +23,37 @@ services:
       - services-tier
     ports:
       - 127.0.0.1:9200:9200
+    healthcheck:
+      test: curl http://127.0.0.1:9200/_cat/health
     environment:
       'discovery.type': 'single-node'
+      'bootstrap.memory_lock': 'true'
+      'ES_JAVA_OPTS': '-Xms512m -Xmx512m'
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+    volumes:
+      - prism-elastic-vol:/usr/share/elasticsearch/data
 
   prism-node:
     container_name: prism-node
     restart: always
     depends_on:
-      - prism-mongo
-      - elasticsearch
+      prism-mongo:
+        condition: service_healthy
+      elasticsearch:
+        condition: service_healthy
     build:
       context: .
       dockerfile: Dockerfile
     networks:
       - services-tier
     ports:
       - $GLS_CONNECTOR_HOST:$GLS_CONNECTOR_PORT:$GLS_CONNECTOR_PORT
+      - $GLS_METRICS_HOST:$GLS_METRICS_PORT:$GLS_METRICS_PORT
+    environment:
+      - NODE_OPTIONS=$GLS_PRISM_NODE_OPTIONS
     env_file:
       - .env
 
@@ -58,6 +73,7 @@ services:
 
 volumes:
   prism-mongodb-vol:
+  prism-elastic-vol:
 
 networks:
   services-tier:
diff --git a/docker-compose.multi-node.yml b/docker-compose.multi-node.yml
@@ -0,0 +1,102 @@
+version: '2.1'
+services:
+  prism-mongo:
+    container_name: prism-mongo
+    image: mongo
+    restart: always
+    healthcheck:
+      test: mongo --eval 'db.getCollectionNames()'
+      interval: 10s
+      timeout: 2s
+      retries: 10
+    volumes:
+      - prism-mongodb-vol:/data/db
+    ports:
+      - 127.0.0.1:27017:27017
+    networks:
+      - services-tier
+
+  elasticsearch:
+    container_name: elasticsearch
+    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.1
+    networks:
+      - services-tier
+    ports:
+      - 127.0.0.1:9200:9200
+    healthcheck:
+      test: curl http://127.0.0.1:9200/_cat/health
+    environment:
+      'discovery.type': 'single-node'
+      'bootstrap.memory_lock': 'true'
+      'ES_JAVA_OPTS': '-Xms512m -Xmx512m'
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+    volumes:
+      - prism-elastic-vol:/usr/share/elasticsearch/data
+
+  prism-node:
+    container_name: prism-node
+    restart: always
+    depends_on:
+      prism-mongo:
+        condition: service_healthy
+    build:
+      context: .
+      dockerfile: Dockerfile
+    networks:
+      - services-tier
+    ports:
+      - $GLS_CONNECTOR_PRISM_HOST:$GLS_CONNECTOR_PRISM_PORT:$GLS_CONNECTOR_PRISM_PORT
+      - $GLS_METRICS_PRISM_HOST:$GLS_METRICS_PRISM_PORT:$GLS_METRICS_PRISM_PORT
+    environment:
+      - NODE_OPTIONS=$GLS_PRISM_NODE_OPTIONS
+      - GLS_ENABLE_BLOCK_HANDLE=true
+      - GLS_ENABLE_PUBLIC_API=false
+      - GLS_CONNECTOR_PORT=$GLS_CONNECTOR_PRISM_PORT
+    env_file:
+      - .env
+
+  prism-api-node:
+    container_name: prism-api-node
+    restart: always
+    depends_on:
+      prism-mongo:
+        condition: service_healthy
+      elasticsearch:
+        condition: service_healthy
+    build:
+      context: .
+      dockerfile: Dockerfile
+    networks:
+      - services-tier
+    ports:
+      - $GLS_CONNECTOR_HOST:$GLS_CONNECTOR_PORT:$GLS_CONNECTOR_PORT
+      - $GLS_METRICS_HOST:$GLS_METRICS_PORT:$GLS_METRICS_PORT
+    environment:
+      - GLS_ENABLE_BLOCK_HANDLE=false
+      - GLS_ENABLE_PUBLIC_API=true
+    env_file:
+      - .env
+
+  mongodb-exporter:
+    container_name: mongodb-exporter
+    image: cyberway/mongodb-exporter:v0.6.2
+    restart: always
+    depends_on:
+      prism-mongo:
+        condition: service_healthy
+    ports:
+      - 9216:9216
+    environment:
+      - MONGODB_URI=mongodb://prism-mongo:27017
+    networks:
+      - services-tier
+
+volumes:
+  prism-mongodb-vol:
+  prism-elastic-vol:
+
+networks:
+  services-tier:
-Original file line number
+Diff line change
@@ Expand Up / @@ -4,3 +4,5 @@ db @@
     .env
     .vscode
     .npmrc
+    docker-compose.yml
+    stats.txt