Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes for tutorial extraction #253

Merged
merged 2 commits into from
Dec 5, 2024
Merged

Fixes for tutorial extraction #253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1a5119a
Add True to video metadata if recordings
bebatut Dec 5, 2024
357a77a
Move tutorial fetch to different job in fetch GH workflow
bebatut Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 32 additions & 6 deletions .github/workflows/fetch_filter_resources.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,9 +70,34 @@ jobs:
with:
name: tools-${{ matrix.subset }}
path: communities/all/resources/${{ matrix.subset }}_tools.*
fetch-tutorials:
runs-on: ubuntu-20.04
name: Fetch tutorials
steps:
- name: Checkout main
uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install requirement
run: |
python -m pip install -r requirements.txt
sudo apt-get install jq
- name: Fetch all tutorials
run: |
bash sources/bin/extract_all_tutorials.sh
env:
PLAUSIBLE_API_KEY: ${{ secrets.PLAUSIBLE_API_TOKEN }}
- name: Archive tutorials artifacts
uses: actions/upload-artifact@v4
with:
name: tutorials
path: communities/all/resources/tutorials.json
merge-fetch-filter:
runs-on: ubuntu-20.04
needs: fetch-tools-stepwise
needs:
- fetch-tools-stepwise
- fetch-tutorials
name: Merge tools, fetch tutorials and filter the resources for communities
steps:
- name: Checkout main
Expand All @@ -90,6 +115,12 @@ jobs:
pattern: tools-*
merge-multiple: true
path: communities/all/resources/
- name: Download stepwise tool lists
uses: actions/download-artifact@v4
with:
pattern: tutorials
merge-multiple: true
path: communities/all/resources/
- name: Display structure of downloaded files
run: ls -R communities/all/resources/
- name: Merge all tools
Expand All @@ -107,11 +138,6 @@ jobs:
- name: Filter workflows for communities
run: |
bash sources/bin/get_community_workflows.sh
- name: Fetch all tutorials
run: |
bash sources/bin/extract_all_tutorials.sh
env:
PLAUSIBLE_API_KEY: ${{ secrets.PLAUSIBLE_API_TOKEN }}
- name: Filter tutorials for communities
run: |
bash sources/bin/get_community_tutorials.sh
Expand Down
2 changes: 2 additions & 0 deletions sources/bin/extract_gtn_tutorials.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,10 @@ def get_youtube_stats(tuto: dict) -> None:
recordings = []
if "recordings" in tuto and tuto["recordings"]:
recordings = tuto["recordings"]
tuto["video"] = True
elif "slides_recordings" in tuto and tuto["slides_recordings"]:
recordings = tuto["slides_recordings"]
tuto["video"] = True
tuto["video_versions"] = len(recordings)
for v in recordings:
url = f"https://www.youtube.com/watch?v={v['youtube_id']}"
Expand Down