Docker image broken with bullseye

[willcohen]

Since the base image has moved to bullseye, a range of things have broken:

• Python has moved to 3.9, so the apt requirements have changed.
• libpython needs to be upgraded because of python 3.9.
• tech.ml.dataset also needs to be upgraded because of libpython, changing all of the internal uses of tech.v2 to tech.v3, which seems to cascade a number of other changes.

Additionally, SELinux requires a :z suffix to the mounted volume to get run.sh to work correctly (podman), which shouldn't be an issue with Docker either. I'll polish up these changes and put a PR together once I get everything working.

[metasoarous]

Hey @willcohen! Thanks so much for pushing this forward!

I have been meaning to update the libpython & tech stacks to take advantage of all the new goodness there (especially excited to be able to expose the clojure api to python notebooks!), and would immensely appreciate your help with that. I had a sense that there might be a bit of churn in the process, so I've been putting it off, but now's a great time to tackle it.

Please let me know if you hit any speed bumps with the process.

Thanks again!

[willcohen]

Sounds good. FWIW -- since I figure my life is going to be a little more docker-centric for a while at least, I bit the bullet and upgraded the linux box I can use to Fedora Silverblue, to just get more comfortable with containering stuff in general. I can wiggle some things around to still use the Docker daemon if need be, but I'll probably be putting podman compatibility through its paces as I start interacting with the dev parts of this project overall more, since the distro pushes me in that direction.

[metasoarous]

Good to hear 👍

It's definitely still helpful to get something like podman working, due to folks having had issues with docker, and some displeasure from folks regarding the direction of docker-desktop. So I approve of and appreciate those efforts.

Thanks again!

[willcohen]

Only a moderate speedbump -- I figure it's better to maybe try to get the existing build working on buster than to try to change everything at once. That said, I cannot for the life of me get umap to import:

CIDER:

2. Unhandled clojure.lang.Compiler$CompilerException                                               
   Error compiling polis/math.clj at (264:17)                                                      
   #:clojure.error{:phase :compile-syntax-check,                                                   
                   :line 264,                                                                      
                   :column 17,                                                                     
                   :source "polis/math.clj"}                                                       
1. Caused by java.lang.RuntimeException                                                            
   No such namespace: umap                                                                         
                                                                                                   
                 Util.java:  221  clojure.lang.Util/runtimeException

The REPL:

21-10-19 16:44:52 af9796f1a0c5 DEBUG [io.netty.util.internal.InternalThreadLocalMap:76] - -Dio.n$ 
 21-10-19 16:44:52 af9796f1a0c5 DEBUG [io.netty.util.internal.InternalThreadLocalMap:76] - -Dio.n$ 
 Syntax error compiling at (math.clj:264:17).                                                      
 No such namespace: umap                                                                           
 user> 

Internet searches suggest that umap can be finicky depending on its package dependencies, which makes this seem a little like #2. Other internet searches suggest that conda is a way around this, but even installing conda, creating a py3.6 environment and using as many prebuilt packages as possible leads to the same error.

Is there any chance you can dig up your last-working local version of the built image? This may well require manually pinning all the python package versions to your known working config. From there I can try to move things toward bullseye and tech.ml v3.

I stand corrected, per https://umap-learn.readthedocs.io/en/latest/:
conda needs conda install -c conda-forge umap-learn to install the prebuilt version. Still some more package errors, but this particular issue is resolved.

[metasoarous]

Excellent; Thanks for the update!

Glad that you got through the umap issue. Please continue to keep me posted!

[maedoc]

any news? I tried to up this today and hit maybe another issue with a changed base image

 % docker-compose -f docker-compose.yml -f docker-compose.dev.yml up
Building analysis
[+] Building 4.4s (6/29)                                                                                                                                            
 => [internal] load build definition from Dockerfile                                                                                                           0.0s
 => => transferring dockerfile: 37B                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                              0.0s
 => => transferring context: 35B                                                                                                                               0.0s
 => [internal] load metadata for docker.io/library/clojure:openjdk-11-tools-deps                                                                               3.9s
 => CACHED [ 1/25] FROM docker.io/library/clojure:openjdk-11-tools-deps@sha256:9d5d25d08129f2f3990ee099500bfb7d8f2de583a76ed5d4563dd23fc27df709                0.0s
 => [internal] load build context                                                                                                                              0.0s
 => => transferring context: 2.69kB                                                                                                                            0.0s
 => ERROR [ 2/25] RUN echo "deb http://fr.archive.ubuntu.com/ubuntu bionic main" >> /etc/apt/sources.list &&    apt-key adv --keyserver keyserver.ubuntu.com   0.4s
------
 > [ 2/25] RUN echo "deb http://fr.archive.ubuntu.com/ubuntu bionic main" >> /etc/apt/sources.list &&    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32 &&    apt-get update &&    apt-get install libpython3.6-dev python3-pip -y --allow-unauthenticated:
#5 0.332 E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

tho it may be unrelated, and I could file a separate issue.

[willcohen]

I think it's a separate issue. My immediate deadline to try to work on this passed a little bit but I would still like to solve it. I'm going down a small rabbit-hole about Docker issues etc in the meantime so I'm not going so blindly here!