PermissionError: oaklib install fails in ODK container (Pystow)

[joeflack4]

Overview

I am trying to our slurp that was previously succeeding, but after a recent OAK release, it no longer works. It looks like Docker is trying to install oaklib, and it fails there.

Error

ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/.local' Check the permissions.

log.txt

Possible solutions

a. Ditch pystow for some other caching method.
b. Can pystow cache stuff somewhere else that would alleviate the issue? Such as within the virtual environment, rather than a user data directory?
c. Provide the user some way to put in the directory (not sure if this will work if failing on pip install)

Additional info

I don't think the details of the command are needed here, as it seems to be failing on pip install. But FYI, in mondo-ingest, from src/ontology/, I'm running export ODK_TAG=dev; sh run.sh make slurp/ordo.tsv (source).

Nico had some comments here: cthoyt/pystow#10

[joeflack4]

@matentzn I meant to make this issue a week ago. Tagging you as you may have comments.

[cmungall]

Did you mean to make an issue in this repo? This looks like it relates to another project.

I'm closing this for now, but if you think this is an OAK issue, make a new issue describing steps to reproduce the problem.

[joeflack4]

@cmungall Yes, Nico asked me to open this issue here. I'm not sure when OAK added PyStow, but this ODK PermissionError issue has been happening to us ever since, and it appears to fail on installing oaklib.

As for solutions, (b) and (c) may be possible for OAK to do if PyStow is highly configurable, and depending on how oaklib's installation works, but I don't know enough to say. My guess is probably not, but I wanted to throw them out there anyway.

Until pystow addresses (and OAK upgrades it) or OAK works around it, we'll be unable to run the slurp pipeline in mondo-ingest from within ODK.

If you need steps to reproduce, refer to what I wrote in "Additional info" (shoulda called it "steps to reproduce").

I can open a similar issue in PyStow as well @cmungall @matentzn , if it helps solve our ODK issue faster.

[matentzn]

The problem was that pystow tried to write .data into a directory somewhere that was not writable. Maybe that is not a problem anymore.

[cmungall]

Yes pystow assumes by default a particular location. Maybe here we want a new issue for an enhancement to allow pystow to be globally configured? For now of course you can mount ~/.data

…

On Sat, Dec 17, 2022 at 8:33 PM Nico Matentzoglu ***@***.***> wrote: The problem was that pystow tried to write .data into a directory somewhere that was not writable. Maybe that is not a problem anymore. — Reply to this email directly, view it on GitHub <#416 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMMOJ3OZA6NTYW455PLODWN2H2TANCNFSM6AAAAAATBRM3VA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cmungall]

Oh I see you already filed an issue cthoyt/pystow#10

I guess unless this is fixed the options are to mount or implement a more configurable pystow like mechanism in OAK?/semsql? I don't think that would be so hard we don't need most of the pystow functionality. But it probably wouldn't be hard to make a PR on pystow either

[cmungall]

You do want to think about your overall strategy here. The ready made SQLite files are potentially large. You might want to be in more control of when they are downloaded, where they cached, when they are refreshed. This sounds like a data ops thing upstream of oak. Eg have your pipeline download the SQLite, deploy it somewhere visible in the docker container, and then just point to the .db file when running oak. This bypasses pystow

[matentzn]

I dont have any opinions on this matter beyond the fact that oak shouldn't fail hard when ~/ happens to not be writable. In all of my use cases, where oak is a part of the make workflow, I do not need anyone to cache my intermediates - I do that myself!

[cmungall]

The alternative to failing hard is to do something clever, or to make opinionated choices. For example "hmm, I seem to be unable to cache this, therefore I am going to do a fresh download each time". Or: "I am going to cycle through some alternative choices of where to cache this". IMO these are bad options as they lead to unexpected hard to debug behavior.

I am a huge proponent of hard fail with informative errors vs silent magic (provided the hard fail is caused by a genuine issue the framework is not expected to solve, e.g. parse error, network error, permission error in a documented location)

We do need the errors to be more informative. Solving this kind of thing in the general case is hard though.

It's hard to anticipate all exceptions thrown by any external library (reuse is good but it increase the attack surface area for unexpected things)

it's hard to handle all exceptions gracefully. Developers complain if there is insufficient information in the error message. Non-developers complain when given large geeky stacktraces. We give people the option of how much information to show with --stacktrace and -v and -vv etc but people get surprised if the default is different from what they would have chosen.

[joeflack4]

Thanks for these thoughts and suggestions, Chris. It looks like Nico was able to resolve this. I checked the git diff on run.sh. Maybe I'm looking in the wrong place. Still not seeing that he did it, but glad it's been done.

[matentzn]

The problem was that I was trying to use my Unix user inside the docker container, and that Unix user does not have a home directory inside the docket container.

[joeflack4]

I understand that, but I just don't know what you changed to fix that. Not critical that I need to know those details though

[matentzn]

Look at contents of run.sh file: I simply removed the -u parameter from the docker run command.

[cthoyt]

I'm not sure what the issue is here, PyStow can be globally configured in one of several ways: https://github.com/cthoyt/pystow#%EF%B8%8F%EF%B8%8F-configuration. Are none of these options sufficient?

I noticed that these aren't included in the readthedocs so i will update that