docker model model-exahype-tsunami errors on Apple Silicon mac

[cmarcotte]

Running docker run -it -p 4242:4242 linusseelinger/model-exahype-tsunami in tandem with python3 qmcpy-client.py http://localhost:4242 yields the following output:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Running on number of ranks: 1
Listening on port 4242...
Reading options
Entered for level 0

And eventually a looping (presumably as a response to repeated model calls) error sequence:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
qemu: uncaught target signal 6 (Aborted) - core dumped
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node 07516088c83b exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Exahype exit status 34304
Outputs read from exahype: 
0
0
0
0
Left
Reading options
Entered for level 0

This is using the AS-native Docker Desktop application, which may play a role, because the Intel application will not run under Rosetta 2 (x86_64 emulation).

[linusseelinger]

@cmarcotte Sorry for my late reply! We have made a bunch of changes to the tsunami model that should make it more reliable in cases like yours. Is the issue still there?

[cmarcotte]

Sorry for the delay: can confirm the error persists.

Entered for level 0
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
qemu: uncaught target signal 6 (Aborted) - core dumped
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node b23f0287cac1 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Exahype exit status 34304
Outputs read from exahype: 
0
0
0
0
Left
[repeats...]

[annereinarz]

@cmarcotte @chun9l thinks adding --platform linux/amd64 to your docker run command might fix the issue, neither of us have a machine to test that on. Would you mind testing?

[chun9l]

Got the idea from this thread: https://stackoverflow.com/questions/66662820/m1-docker-preview-and-keycloak-images-platform-linux-amd64-does-not-match-th

[chun9l]

Erm might be a simpler fix than I initially thought. The image in the Dockerfile is openmpi-builder:0.3.0 that has no arm64 build according to their dockerhub (https://hub.docker.com/r/mpioperator/openmpi-builder/tags). Using a newer image may fix the issue.

[chun9l]

@cmarcotte I've pushed an image compiled for Arm to my DockerHub. Could you please replace mpioperator with chun9l on the first line of the Dockerfile and check if it builds?

[cmarcotte]

First: sorry I missed the discussion.

@chun9l Using your arm image, it now fails due to HDF5. Adding --disable-hdf5 to the config line (16) let's it progress to failure on building the tsunami model (lines 56-62). Adding --platform linux/amd64 to the docker build command allows it to build the tsunami model. Then it failed because I didn't have server.cpp in the directory -- correcting lets the build finish.

The successful build command is:

▶ docker buildx build . --platform linux/amd64 -t stupefied_napier        

However, running the container (with or without the --platform linux/amd64 option) leads to the same old error:

▶ docker run -it -p 4242:4242 -v ~/tsunami_output:/output --platform linux/amd64 stupefied_napier
Running on number of ranks: 1
Listening on port 4242...
Reading options
Entered for level 0
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node 7cb9baca43fb exited on signal 4 (Illegal instruction).
--------------------------------------------------------------------------
Exahype exit status 33792
Outputs read from exahype: 
0
0
0
0
Left

While on the python side:

▶ python test_output.py http://localhost:4242
Connecting to host URL http://localhost:4242
[0.0, 0.0, 0.0, 0.0]
Traceback (most recent call last):
  File "/Users/christophermarcotte/Downloads/test_output.py", line 19, in <module>
    assert pytest.approx(output[0], 18) == 1847.590542295737578, "Tsunami output not as expected"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Tsunami output not as expected

[chun9l]

Hmm that's interesting because I also got the HDF5 error if I force my laptop (with Intel CPU) to build the arm64 image. But the amd64 one passes as usual. Looks like an issue with the apt packages then...

Ideally, Docker should auto detect the cpu architecture and pull the correct image. So the --platform flag is not needed. Perhaps the ExaHyPE build failed because of HDF5? I'll delegate this to @annereinarz ;)

Thanks for trying it out nevertheless!

[chun9l]

I found the problem: some paths were hardcoded to the x86 architecture. The problem with HDf5 is gone after fixing it, but it's taking a while to finish building. I've opened #89 for this.

[chun9l]

More trouble unfortunately:(

It is now complaining

[Errno 2] No such file or directory: '/ExaHyPE-Tsunami/Submodules/libxsmm/bin/libxsmm_gemm_generator': '/ExaHyPE-Tsunami/Submodules/libxsmm/bin/libxsmm_gemm_generator'

which I suspect is due to the library failing to build (due to change in CPU arch) from the command (lines 56-62)

    ../../Toolkit/toolkit.sh SWE_asagi_limited_l0.exahype2 && \

[annereinarz]

Ah, that makes sense. libxsmm is only supported on intel cpus. It works on the newer amd ones as well, but I think that is just because of the similar architecture. We can switch off the optimised kernels in which case it no longer uses libxsmm but that would make the solver considerably slower. I think we should introduce a flag which switches this on and pass it through as an optional configuration option. I can try that next week. If you want to give it a go you can change the configuration file:

• arch should be noarch instead of sandybridge
• optmised kernels should be false

[chun9l]

Just tested the newer libxsmm, it also works for Arm. We can add a line in the Dockerfile to clone and build it.
What is the best way to change the arch in the config file? I want to avoid grep and replace if possible...

[linusseelinger]

In addition to making the dockerfiles ARM compatible, we'll also have to modify our CI builds to include ARM. Opened an issue #90 . Probably have to wait a few months till ARM runners are supposed to become available

[chun9l]

#89 should enable arm64 build from Dockerfile now.