Merge pull request #9 from NelsonDane/master

Update Endpoints, Add Private Repo Support, Variable Delay Time and Update Readme

.gitignore

@@ -1,3 +1,4 @@
 .idea/
 node_modules/
 .secrets.rc
+.vscode/

Dockerfile

@@ -3,5 +3,6 @@ FROM node:lts-alpine
 WORKDIR /app
 COPY package.json package-lock.json ./
 RUN npm ci
-COPY . .
+COPY docker-entrypoint.sh .
+COPY src ./src
 CMD [ "/app/docker-entrypoint.sh" ]

README.md

@@ -1,58 +1,110 @@
-# Mirror your github repositories to your gitea server
+# Automatically Mirror Github Repo To Your Gitea Server
 
 ## Badges
 
 [![image pulls](https://img.shields.io/docker/pulls/jaedle/mirror-to-gitea.svg)](https://cloud.docker.com/repository/docker/jaedle/mirror-to-gitea)
-[![microbadger analysis](https://images.microbadger.com/badges/image/jaedle/mirror-to-gitea.svg)](https://microbadger.com/images/jaedle/mirror-to-gitea "Get your own image badge on microbadger.com")
 
 ## Description
 
-This script mirrors automatically the public repositories from a github-user or github-organization to your gitea server.
-It will - once started - create a mirrored repository under a given token for a gitea user fully automatically.
+This script automatically mirrors the repositories from a github-user or github-organization to your gitea server.
+Once started, it will create a mirrored repository under a given token for a gitea user, completely automatically.
 
 Example:
 A github user `github-user` has public repositories `dotfiles` and `zsh-config`.
-Starting the script with a gitea token for the account `gitea-user` will create the following mirror repositories:
+Starting the script with a gitea token for the account `gitea-user` will create the following mirrored repositories:
 
-- github.com/github-user/dotfiles ← some-gitea.url/gitea-user/dotfiles
-- github.com/github-user/zsh-config ← some-gitea.url/zsh-config/dotfiles
+- github.com/github-user/dotfiles → your-gitea.url/gitea-user/dotfiles
+- github.com/github-user/zsh-config → your-gitea.url/gitea-user/zsh-config
 
 The mirror settings are default by your gitea instance.
 
+It is also possible to mirror private repos, which can be configred here in [#parameters](#parameters). When mirroring private repos, they will be created as private repos on your gitea server.
+
 ## Prerequisites
 
-- Something to mirror (a github user or organization with public repos)
-- Gitea instance up and running
-- User for Gitea with generated token
-- Docker
+- A github user or organization with repositories
+- Configured Gitea instance up and running
+- User for Gitea with generated token (Settings -> Applications -> Generate New Token)
+- Docker or Docker Compose
 
-## Run it
+### Docker Run
 
 ```sh
-docker container run \
+docker run \
  -d \
  --restart always \
  -e GITHUB_USERNAME=github-user \
- -e GITEA_URL=https://some-gitea.url \
+ -e GITEA_URL=https://your-gitea.url \
  -e GITEA_TOKEN=please-exchange-with-token \
  jaedle/mirror-to-gitea:latest
 ```
 
-This will a spin up a docker container running infinite which will try to mirror all your repositories once every hour to your gitea server.
+This will a spin up a docker container which will run forever, mirroring all your repositories once every hour to your gitea server.
+
+### Docker Compose
+
+```yaml
+version: "3.3"
+services:
+    mirror-to-gitea:
+        image: jaedle/mirror-to-gitea:latest
+        restart: always
+        environment:
+          - GITHUB_USERNAME=github-user
+          - GITEA_URL=https://your-gitea.url
+          - GITEA_TOKEN=please-exchange-with-token
+          #- GITHUB_TOKEN=please-exchange-with-token # Optional, set to mirror private repos
+          #- MIRROR_PRIVATE_REPOSITORIES=true # Optional, set to mirror private repos
+          # - DELAY=3600 # Optional, set to change the delay between checks (in seconds)
+        container_name: mirror-to-gitea
+```
+## Building from Source
+
+### Prerequisites
+- NodeJS
+- NPM
+
+### Build
+```sh
+npm install
+```
+If errors occur, try deleting the `package-lock.json` file and run `npm install` again.
+
+### Build Docker Image
+```sh
+docker build -t mirror-to-gitea .
+```
+
+### Run With NodeJS
+```sh
+export GITHUB_USERNAME=github-user
+export GITEA_URL=https://your-gitea.url
+export GITEA_TOKEN=please-exchange-with-token
+node src/index.js
+```
+Also export `GITHUB_TOKEN` and `MIRROR_PRIVATE_REPOSITORIES` if you want to mirror private repos, or `DELAY` if you want to change the delay between checks.
+
+### Run With Docker
+In the above Docker run command, replace `jaedle/mirror-to-gitea:latest` with `mirror-to-gitea`.
+In your Docker Compose file, replace `jaedle/mirror-to-gitea:latest` with `build: .`. Then run `docker compose build` and `docker compose up -d`.
+
+## Parameters
 
-### Parameters
+### Required
+- `GITHUB_USERNAME`: The name of your user or organization which public repos should be mirrored
+- `GITEA_URL`: The url of your gitea server
+- `GITEA_TOKEN`: The token for your gitea user (Settings -> Applications -> Generate New Token)
 
-- `GITHUB_USERNAME` name of user or organization which public repos should be mirrored
-- `GITHUB_TOKEN` [GitHub personal access token](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token) (optional)
-- `GITEA_URL` url of your gitea server
-- `GITEA_TOKEN` token for your gitea user
+### Optional
+- `GITHUB_TOKEN`: [GitHub personal access token](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token). **Attention: if this is set, the token will be transmitted to your specified Gitea instance!**
+- `MIRROR_PRIVATE_REPOSITORIES`: If set to `true`, your private GitHub repositories will also be mirrored to gitea. The `GITHUB_TOKEN` parameter must be set for this to work.
+- `DELAY`: How often to check for new repositories in seconds. Default is 3600 (1 hour).
 
 ## Things to do
 
-- refactoring
-- think about how to test
-- configurable interval
-- better logging
-- use github token to solve problems with rate limits
-- add gitlab support
-- and so on..
+- Refactoring
+- Think about how to test
+- Better logging
+- Use github token to solve problems with rate limits
+- Add gitlab support
+- And so on..

docker-entrypoint.sh

@@ -2,13 +2,14 @@
 
 set -e
 
-ONE_HOUR_DELAY=3600
+# Get custom delay, else use 3600 seconds
+DELAY="${DELAY:-3600}"
 
 while true
 do
-  echo 'Starting to create mirrors'
+  echo "Starting to create mirrors..."
   node /app/src/index.js
 
-  echo 'Waiting...'
-  sleep "${ONE_HOUR_DELAY}"
+  echo "Waiting for ${DELAY} seconds..."
+  sleep "${DELAY}"
 done

src/index.js

@@ -3,24 +3,49 @@ const request = require('superagent');
 const {default: PQueue} = require('p-queue');
 
 
-async function getGithubRepositories(username, token) {
+async function getGithubRepositories(username, token, mirrorPrivateRepositories) {
   const octokit = new Octokit({
     auth: token || null,
   });
-  return octokit.paginate('GET /users/:username/repos', { username: username })
+
+  const publicRepositoriesWithForks = await octokit.paginate('GET /users/:username/repos', { username: username })
+      .then(repositories => toRepositoryList(repositories));
+
+  let allRepositoriesWithoutForks;
+  if(mirrorPrivateRepositories === 'true'){
+  allRepositoriesWithoutForks = await octokit.paginate('GET /user/repos?visibility=public&affiliation=owner&visibility=private')
     .then(repositories => toRepositoryList(repositories));
+  }
+
+  if(mirrorPrivateRepositories === 'true'){
+    return filterDuplicates(allRepositoriesWithoutForks.concat(publicRepositoriesWithForks));
+  }else{
+    return publicRepositoriesWithForks;
+  }
 }
 
 function toRepositoryList(repositories) {
   return repositories.map(repository => {
-    return { name: repository.name, url: repository.clone_url }
+    return { name: repository.name, url: repository.clone_url, private: repository.private };
   });
 }
 
+function filterDuplicates(array) {
+  var a = array.concat();
+  for(var i=0; i<a.length; ++i) {
+      for(var j=i+1; j<a.length; ++j) {
+          if(a[i].url === a[j].url)
+              a.splice(j--, 1);
+      }
+  }
+
+  return a;
+}
+
 async function getGiteaUser(gitea) {
   return request.get(gitea.url
     + '/api/v1/user')
-    .query(`access_token=${gitea.token}`)
+    .set('Authorization', 'token ' + gitea.token)
     .then(response => {
       return { id: response.body.id, name: response.body.username }
     });
@@ -30,19 +55,21 @@ function isAlreadyMirroredOnGitea(repository, gitea, giteaUser) {
   const requestUrl = `${gitea.url}/api/v1/repos/${giteaUser.name}/${repository}`;
   return request.get(
     requestUrl)
-    .query(`access_token=${gitea.token}`)
+    .set('Authorization', 'token ' + gitea.token)
     .then(() => true)
     .catch(() => false);
 }
 
-function mirrorOnGitea(repository, gitea, giteaUser) {
+function mirrorOnGitea(repository, gitea, giteaUser, githubToken) {
   request.post(`${gitea.url}/api/v1/repos/migrate`)
-    .query(`access_token=${gitea.token}`)
+    .set('Authorization', 'token ' + gitea.token)
     .send({
+      auth_token: githubToken || null,
       clone_addr: repository.url,
       mirror: true,
       repo_name: repository.name,
       uid: giteaUser.id,
+      private: repository.private
     })
     .then(() => {
       console.log('Did it!');
@@ -53,15 +80,15 @@ function mirrorOnGitea(repository, gitea, giteaUser) {
 
 }
 
-async function mirror(repository, gitea, giteaUser) {
+async function mirror(repository, gitea, giteaUser, githubToken) {
   if (await isAlreadyMirroredOnGitea(repository.name,
     gitea,
     giteaUser)) {
     console.log('Repository is already mirrored; doing nothing: ', repository.name);
     return;
   }
   console.log('Mirroring repository to gitea: ', repository.name);
-  await mirrorOnGitea(repository, gitea, giteaUser);
+  await mirrorOnGitea(repository, gitea, giteaUser, githubToken);
 }
 
 async function main() {
@@ -72,6 +99,7 @@ async function main() {
   }
   const githubToken = process.env.GITHUB_TOKEN;
   const giteaUrl = process.env.GITEA_URL;
+
   if (!giteaUrl) {
     console.error('No GITEA_URL specified, please specify! Exiting..');
     return;
@@ -83,8 +111,14 @@ async function main() {
     return;
   }
 
+  const mirrorPrivateRepositories = process.env.MIRROR_PRIVATE_REPOSITORIES;
+  if(mirrorPrivateRepositories === 'true' && !githubToken){
+    console.error('MIRROR_PRIVATE_REPOSITORIES was set to true but no GITHUB_TOKEN was specified, please specify! Exiting..')
+    return;
+  }
+
 
-  const githubRepositories = await getGithubRepositories(githubUsername, githubToken);
+  const githubRepositories = await getGithubRepositories(githubUsername, githubToken, mirrorPrivateRepositories);
   console.log(`Found ${githubRepositories.length} repositories on github`);
 
   const gitea = {
@@ -96,7 +130,7 @@ async function main() {
   const queue = new PQueue({ concurrency: 4 });
   await queue.addAll(githubRepositories.map(repository => {
     return async () => {
-      await mirror(repository, gitea, giteaUser);
+      await mirror(repository, gitea, giteaUser, githubToken);
     };
   }));
 }