Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLD: Refresh gitpod build files #56375

Closed
wants to merge 11 commits into from
Closed

BLD: Refresh gitpod build files #56375

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3cc22d0
Refresh the old dockerfile
lithomas1 Dec 7, 2023
ccf1f0a
Update .gitpod.yml
lithomas1 Dec 7, 2023
85f354f
Update .gitpod.yml
lithomas1 Dec 7, 2023
4b6129d
update workspace files
lithomas1 Dec 7, 2023
c305380
more updates
lithomas1 Dec 7, 2023
f780db9
one more missing permissions
lithomas1 Dec 7, 2023
a3ff3cd
more fixes
lithomas1 Dec 7, 2023
722df46
one more activate
lithomas1 Dec 7, 2023
4b27d44
Update gitpod.Dockerfile
lithomas1 Dec 7, 2023
2342fb6
Update gitpod.Dockerfile
lithomas1 Dec 7, 2023
1835adb
simplify greatly
lithomas1 Dec 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .gitpod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,23 @@
# images for gitpod pandas are in https://hub.docker.com/r/pandas/pandas-gitpod/tags
# we're using the Dockerfile in the base of the repo
image:
file: Dockerfile
file: gitpod/gitpod.Dockerfile
tasks:
- name: Prepare development environment
init: |
mkdir -p .vscode
cp gitpod/settings.json .vscode/settings.json
git fetch --tags
conda init
source ~/.bashrc
conda env update --file environment.yml --prune
conda activate pandas-dev
python -m pip install -ve . --no-build-isolation --config-settings editable-verbose=true
pre-commit install
command: |
source ~/.bashrc
conda activate pandas-dev
conda env update --file environment.yml --prune
python -m pip install -ve . --no-build-isolation --config-settings editable-verbose=true
echo "✨ Pre-build complete! You can close this terminal ✨ "

Expand Down
50 changes: 5 additions & 45 deletions gitpod/gitpod.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,48 +1,8 @@
# Doing a local shallow clone - keeps the container secure
# and much slimmer than using COPY directly or making a
# remote clone
ARG BASE_CONTAINER="pandas/pandas-dev:latest"
FROM gitpod/workspace-base:latest as clone
FROM condaforge/miniforge3:23.3.1-1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One potential issue with using the conda image here is that it is pinned to a specific version. This can lead to Gitpod breaking like in #53685 when the version of conda becomes incompatible with the version of python and other dependencies the environment.yml file.

I think building from the base repo's dockerfile without conda is probably the simpler and easier to maintain approach, as the gitpod instance will update with the base repo's dockerfile. If we were to go down that route, the gitpod folder could actually be deleted entirely, and a .gitpod.yml file is all that is required.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I switched to conda from Docker is that docker doesn't provide a way to install packages (I don't think there is any sudo in the gitpod container - at least apt-get/pip don't work for me).

I think Docker might also have the same problem since the packages installed will be cached and won't update unless you recreate the gitpod instance.

I avoid the problem here by updating the conda packages on install based on the environment.yml. I'm happy to switch back to Docker if there's a way to do the same there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think most Docker images do not include sudo. What error were you seeing when doing something like apt-get update && apt-get install -y <some_package>?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like this

gitpod@lithomas1-pandas-1tcur93d5vv:/workspace/pandas$ apt install build-essential
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
gitpod@lithomas1-pandas-1tcur93d5vv:/workspace/pandas$ sudo apt install build-essential
bash: sudo: command not found

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm strange...and you are running that between the

USER root

....

USER gitpod

commands in the image right? I think anything in between those should be running under the root user account

Copy link
Member Author

@lithomas1 lithomas1 Jan 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is when the workspace is started up, not in the Dockerfile.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the Dockerfile intentionally runs under a non-priviledged user. So you'd have to install system libraries before that or remove the USER gitpod statement in gitpod.Dockerfile (no idea if that is a good idea for gitpod or not)


# the clone should be deep enough for versioneer to work
RUN git clone https://github.com/pandas-dev/pandas --depth 12 /tmp/pandas
# Init conda and use the speedy libmamba solver
RUN conda init && conda install -n base conda-libmamba-solver && conda config --set solver libmamba

# -----------------------------------------------------------------------------
# Using the pandas-dev Docker image as a base
# This way, we ensure we have all the needed compilers and dependencies
# while reducing the build time
FROM ${BASE_CONTAINER} as build
# Install dependencies
RUN conda env update --file https://raw.githubusercontent.com/pandas-dev/pandas/main/environment.yml --prune

# -----------------------------------------------------------------------------
USER root

# -----------------------------------------------------------------------------
# ---- ENV variables ----
# ---- Directories needed ----
ENV WORKSPACE=/workspace/pandas/ \
CONDA_ENV=pandas-dev

# Allows this micromamba.Dockerfile to activate conda environments
SHELL ["/bin/bash", "--login", "-o", "pipefail", "-c"]

# Copy over the shallow clone
COPY --from=clone --chown=gitpod /tmp/pandas ${WORKSPACE}

# Everything happens in the /workspace/pandas directory
WORKDIR ${WORKSPACE}

# Build pandas to populate the cache used by ccache
RUN git config --global --add safe.directory /workspace/pandas
RUN conda activate ${CONDA_ENV} && \
python -m pip install -e . --no-build-isolation && \
python setup.py build_ext --inplace && \
ccache -s

# Gitpod will load the repository into /workspace/pandas. We remove the
# directory from the image to prevent conflicts
RUN rm -rf ${WORKSPACE}

# -----------------------------------------------------------------------------
# Always return to non privileged user
RUN chown -R gitpod:gitpod /home/gitpod/.cache/
USER gitpod
Loading