Skip to content

feat: ✨ add initial collaborate on GenomeDK section #385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
signekb wants to merge 7 commits into
base: main
Choose a base branch
from
+172 −154
Open

feat: ✨ add initial collaborate on GenomeDK section #385

Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f2c3c1d
style: :art: format `qmd`
signekb Jul 7, 2026
1624d35
feat: :sparkles: add initial working in a shared folder on GenomeDK s…
signekb Jul 7, 2026
c654cee
fix: :pencil2: add missing anchor for outside genomedk header
signekb Jul 7, 2026
facc89d
Merge branch 'main' into feat/add-initial-collaborate-on-genomedk-sec…
signekb Jul 7, 2026
c877ded
fix: :pencil2: remove duplicate paragraph from merge
signekb Jul 8, 2026
e854a6d
Merge branch 'main' into feat/add-initial-collaborate-on-genomedk-sec…
signekb Jul 15, 2026
8f9398f
docs: :memo: update collaborating section based on review
signekb Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
280 changes: 160 additions & 120 deletions how-we-work/genome-dk.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,109 +4,156 @@ title: "Working on GenomeDK"

## GenomeDK

[GenomeDK](https://genome.au.dk) is the name of Aarhus University's
super computer cluster. It is a high-performance computing (HPC) system.
It is ISO 27001 certified and approved to store sensitive data in
compliance with GDPR and the Danish Data Protection Act.
[GenomeDK](https://genome.au.dk) is the name of Aarhus University's super
computer cluster. It is a high-performance computing (HPC) system. It is ISO
27001 certified and approved to store sensitive data in compliance with GDPR and
the Danish Data Protection Act.

Within a Seedcase context we are going to use it as an example of how
Sprout can be used to import, organise and store data on a remote
cluster. It will also be used by the two projects that we are currently
working with, [DP-Next](https://dp-next.github.io) and
[ON-LiMiT](https://onlimit.org/).
Within a Seedcase context we are going to use it as an example of how Sprout can
be used to import, organise and store data on a remote cluster. It will also be
used by the two projects that we are currently working with,
[DP-Next](https://dp-next.github.io) and [ON-LiMiT](https://onlimit.org/).

## Access

People associated with a university, the public sector (government
agencies, regions, and municipalities), or small and medium-sized
enterprises (SMEs) can ask for an account on GenomeDK, which is done via
the [request
account](https://console.genome.au.dk/user-requests/create/) portal. For
now, we will need to select the "Open" zone. You need to state your
preferred username and password even though the form seems to say to
only do it if you want to re-activate an old account.
People associated with a university, the public sector (government agencies,
regions, and municipalities), or small and medium-sized enterprises (SMEs) can
ask for an account on GenomeDK, which is done via the [request
account](https://console.genome.au.dk/user-requests/create/) portal. For now, we
will need to select the "Open" zone. You need to state your preferred username
and password even though the form seems to say to only do it if you want to
re-activate an old account.

## Setting up

Once you get approved for access you can run through the
[documentation](https://genome.au.dk/docs/), which is extensive and
quite detailed. Because GenomeDK is a Linux-based server, you setup,
access, and do many actions through a shell terminal. You will have to
set up two-factor authentication using an authenticator app (they don't
stipulate any in particular).
[documentation](https://genome.au.dk/docs/), which is extensive and quite
detailed. Because GenomeDK is a Linux-based server, you setup, access, and do
many actions through a shell terminal. You will have to set up two-factor
authentication using an authenticator app (they don't stipulate any in
particular).

## Connecting

### SSH {#ssh}

You can connect to your GenomeDK environment from the terminal on your
local machine. Follow the
[instructions](https://genome.au.dk/docs/getting-started/#ssh) on
GenomeDK's website.
You can connect to your GenomeDK environment from the terminal on your local
machine. Follow the
[instructions](https://genome.au.dk/docs/getting-started/#ssh) on GenomeDK's
website.

### Desktop {#desktop}

If you prefer a graphical interface, you can also access GenomeDK
through their [virtual desktop](https://desktop.genome.au.dk). Follow
the [instructions](https://genome.au.dk/docs/getting-started/#desktop)
on GenomeDK's website.
If you prefer a graphical interface, you can also access GenomeDK through their
[virtual desktop](https://desktop.genome.au.dk). Follow the
[instructions](https://genome.au.dk/docs/getting-started/#desktop) on GenomeDK's
website.

### Mounting files

If you want to view and edit the files stored on GenomeDK using
applications installed on your local machine, you can mount your
GenomeDK file system locally. Follow the
If you want to view and edit the files stored on GenomeDK using applications
installed on your local machine, you can mount your GenomeDK file system
locally. Follow the
[instructions](https://genome.au.dk/docs/working-with-data/#mounting) on
GenomeDK's website.

## Installing software tools

To develop Seedcase projects on GenomeDK, you will need to install uv.
If you want to render markdown documents, you will also need to install
Quarto. Finally, if you want to run recipes in the
[`justfile`](workflow.qmd#workflow-helper), you will need to install
`just`.
To develop Seedcase projects on GenomeDK, you will need to install uv. If you
want to render markdown documents, you will also need to install Quarto.
Finally, if you want to run recipes in the
[`justfile`](workflow.qmd#workflow-helper), you will need to install `just`.

### Installing uv {#installing-uv}

Log in to GenomeDK (either through [SSH](#ssh) or the
[desktop interface](#desktop)) and open a terminal window. Following
uv's
[installation guide](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer),
Log in to GenomeDK (either through [SSH](#ssh) or the [desktop
interface](#desktop)) and open a terminal window. Following uv's [installation
guide](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer),
run:

``` {.bash filename="Terminal"}
```bash {filename="Terminal"}
curl -LsSf https://astral.sh/uv/install.sh | sh
```

### Installing Quarto

Log in to GenomeDK (either through [SSH](#ssh) or the
[desktop interface](#desktop)) and open a terminal window. Follow the
instructions on Quarto's
Log in to GenomeDK (either through [SSH](#ssh) or the [desktop
interface](#desktop)) and open a terminal window. Follow the instructions on
Quarto's
[website](https://quarto.org/docs/download/tarball.html?version=1.7.32&idPrefix=download).

Notes:

- Step 3: if `~/.local/bin` exists, you don't need to run
`mkdir ~/.local/bin`
- Step 3: if `~/.local/bin` exists, you don't need to run `mkdir ~/.local/bin`
- Step 4: as the instructions say, you can probably skip this step
- If VS Code is open, you'll need to restart it to detect the Quarto CLI

### Installing `just`

Log in to GenomeDK (either through [SSH](#ssh) or the
[desktop interface](#desktop)) and open a terminal window. Following
`just`'s
[installation guide](https://just.systems/man/en/pre-built-binaries.html#pre-built-binaries),
Log in to GenomeDK (either through [SSH](#ssh) or the [desktop
interface](#desktop)) and open a terminal window. Following `just`'s
[installation
guide](https://just.systems/man/en/pre-built-binaries.html#pre-built-binaries),
run:

``` {.bash filename="Terminal"}
```bash {filename="Terminal"}
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to ~/bin
```

## Collaborating and working in a shared folder

When you work in a shared folder on GenomeDK, any changes you make (e.g.,

@lwjohnst86 lwjohnst86 Jul 8, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you include a concrete folder structure here for "shared folder"? Are you working in the exactly the same folder? I made a comment or issue about this I thought, but I can't find it. Do you work like:

onlimit-study/
└── feasibility-data/
    ├── main/
    ├── signe/
    └── marton/

So you have a clone in signe/ and Marton works in marton/? Or do you all work in feasibility-data/? If the second, we should probably work like the above to avoid this "shared editing" situation.

@signekb signekb Jul 8, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes, it's the second situation; there's one feasibility-data/ that everyone works in. With the first, you mean we each have a branch, or that we have n_users + 1 clones of the repo (including the data)?

@lwjohnst86 lwjohnst86 Jul 8, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We each have clones (with our name for the clone), where we all push and pull from GitHub.

@signekb signekb Jul 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we then only have the checked-out data files in the main/ repo? Otherwise, we would have multiple copies of the raw data - one copy for each local repo - right?

@lwjohnst86 lwjohnst86 Jul 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be solved by using the build and release process I wrote about. No data is saved outside of the main repo on genomedk

@signekb signekb Jul 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright. I'll update this with my understanding of how we'll work 👍

@martonvago martonvago Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look at this and wrote a comment.

Just two points here:

  • We don't even need to nest the individual clones under a feasibility-data/ folder, everyone can clone where they want. Or is there some benefit to having the clones in a shared folder?
  • I think Signe is right about the data duplication. If a clone wants to do something with the data, it has to smudge it, and then there will be duplication. If the clones never smudge, that is the same as working outside of GenomeDK and then I'm not sure what the clones on GenomeDK are for.

editing files, or checking out to a different branch) will be visible for all
users. So, we follow these practices when working in a shared folder:

1. Whenever possible, work on your own machine (see [Working outside
GenomeDK](#outside-genomedk)), for example with test data. Only go to
GenomeDK to test on real data later in the development stage.
2. Ensure that you leave the repository on GitHub in a clean state, i.e., don't
leave uncommitted changes hanging. This way, others don't have to worry about
what these changes are, and whether someone is still working on them.

## Developing with VS Code

The best way to work on Seedcase projects in VS Code is to use the VS Code
application on your local machine and connect to GenomeDK via SSH. This allows
you to edit files and run code on GenomeDK as if you were working locally. This
setup uses the [Remote SSH VS Code
extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh),
which works by adding a `.vscode-server` folder in your home directory on
GenomeDK. This is a lightweight backend that allows VS Code to run terminals,
extensions, debuggers, etc., remotely on GenomeDK. If at any point you want to
remove the backend, you can simply delete this folder.

1. Open VS Code on your local machine and install the [Remote SSH VS Code
extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh)

2. Select "Remote-SSH: Connect to Host..." from the Command Palette (`F1`,
`Ctrl+Shift+P`) and enter `<username>@login.genome.au.dk`. Follow the prompts
given to update or create your config file. In the "Remotes Explorer" panel,
you should now see the GenomeDK host listed as `login.genome.au.dk`.

3. Click the GenomeDK SSH entry and enter your GenomeDK password when prompted.
Now, you should be connected to GenomeDK and see the remote file system if
you open up the VS Code terminal.

4. Clone your repository using the VS Code terminal, if you haven't already:

```bash {filename="Terminal"}
cd <.../my/projects>
git clone <repo-url>
cd <repo-name>
```

5. Open your project folder in VS Code by clicking the "Open Folder" button in
the panel on the left-hand side
Comment thread
signekb marked this conversation as resolved.
Outdated

6. Install your project dependencies (you'll need [uv
installed](#installing-uv)):

```bash {filename="Terminal"}
uv sync
```
The best way to work on Seedcase projects in VS Code is to use the VS
Code application on your local machine and connect to GenomeDK via SSH.
This allows you to edit files and run code on GenomeDK as if you were
Comment thread
signekb marked this conversation as resolved.
Outdated
Expand Down Expand Up @@ -161,33 +208,32 @@ point you want to remove the backend, you can simply delete this folder.
panel and type `@recommended` into the search bar. You can install extensions
from here.

8. To be able to commit to GitHub, you will need to set up your Git
credentials. You can do this by running the following commands in the
terminal:
8. To be able to commit to GitHub, you will need to set up your Git credentials.
You can do this by running the following commands in the terminal:

``` {.bash filename="Terminal"}
git config --global user.name "Your Name"
git config --global user.email "you@example.com"
```
```bash {filename="Terminal"}
git config --global user.name "Your Name"
git config --global user.email "you@example.com"
```

Now, you should be able to run scripts in your project using either the
VS Code terminal or the Run/Debug UI, as well as push changes to GitHub.
Now, you should be able to run scripts in your project using either the VS Code
terminal or the Run/Debug UI, as well as push changes to GitHub.

## Using Git LFS on GenomeDK

In Data Packages on GenomeDK, all Parquet data files and data kept in
the `raw/` folder are tracked using [Git LFS](https://git-lfs.com/) with
a local LFS store on GenomeDK itself. This means that the contents of
data files are stored on GenomeDK and only pointers to these files are
uploaded to GitHub. On GenomeDK you can see the contents of the data
files in two places: in your working copy of the Data Package and in a
folder outside the Data Package directory called the LFS store.
In Data Packages on GenomeDK, all Parquet data files and data kept in the `raw/`
folder are tracked using [Git LFS](https://git-lfs.com/) with a local LFS store
on GenomeDK itself. This means that the contents of data files are stored on
GenomeDK and only pointers to these files are uploaded to GitHub. On GenomeDK
you can see the contents of the data files in two places: in your working copy
of the Data Package and in a folder outside the Data Package directory called
the LFS store.

When working with data or adding new data to the Data Package, it is
important to take care that the Git LFS configuration is not changed and
that sensitive data is always tracked by Git LFS. Any data that is not
tracked by Git LFS runs the risk of being uploaded to GitHub
accidentally, which would constitute a serious data breach.
When working with data or adding new data to the Data Package, it is important
to take care that the Git LFS configuration is not changed and that sensitive
data is always tracked by Git LFS. Any data that is not tracked by Git LFS runs
the risk of being uploaded to GitHub accidentally, which would constitute a
serious data breach.

### Setting up Git LFS

Expand All @@ -198,80 +244,74 @@ or via
Then run `git lfs install`.

::: callout-important
If you don't initialise Git LFS with `git lfs install`, Git LFS will
not manage data files. Instead, files that should be tracked by LFS
will be handled by Git as normal and uploaded to GitHub on push.
If you don't initialise Git LFS with `git lfs install`, Git LFS will not manage
data files. Instead, files that should be tracked by LFS will be handled by Git
as normal and uploaded to GitHub on push.
:::

Afterwards, create a folder on GenomeDK that everyone working on the
Data Package can access and edit: this will be the LFS store where all
the data will be saved. This folder should be **outside** the Data
Package folder.
Afterwards, create a folder on GenomeDK that everyone working on the Data
Package can access and edit: this will be the LFS store where all the data will
be saved. This folder should be **outside** the Data Package folder.

Set the LFS store folder up as a
[bare repository](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-barerepository)
Set the LFS store folder up as a [bare
repository](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-barerepository)
by opening it in the Terminal and running `git init --bare`.

Then, create an `.lfsconfig` file in the Data Package root folder and
add the absolute path to the LFS store:
Then, create an `.lfsconfig` file in the Data Package root folder and add the
absolute path to the LFS store:

``` {.txt filename=".lfsconfig"}
```txt {filename=".lfsconfig"}
[lfs]
url = file:///path/to/.../lfs-store
```

::: callout-tip
Be sure to get the real filesystem path to this folder, not the one
pointing to your user space. You can find this by running
`readlink -f .` from the the LFS store folder.
Be sure to get the real filesystem path to this folder, not the one pointing to
your user space. You can find this by running `readlink -f .` from the the LFS
store folder.
:::

Define a rule for tracking all Parquet files and all data in the `raw/`
folder by running the following commands from the root of the Data
Package:
Define a rule for tracking all Parquet files and all data in the `raw/` folder
by running the following commands from the root of the Data Package:

``` {.bash filename="Terminal"}
```bash {filename="Terminal"}
git lfs track "*.parquet"
git lfs track "raw/**"
```

This will create a `.gitattributes` file listing the tracked paths.
Tracked files shouldn't be listed in `.gitignore`.
This will create a `.gitattributes` file listing the tracked paths. Tracked
files shouldn't be listed in `.gitignore`.

To get an overview of the LFS settings in the repo, run
`git lfs env` or `git lfs ls-files` from the root of the Data
Package.
To get an overview of the LFS settings in the repo, run `git lfs env` or
`git lfs ls-files` from the root of the Data Package.

### Adding data to LFS

1. To add data, create a new Parquet file or move data file(s) into the
`raw/` folder, which is tracked by LFS.
1. To add data, create a new Parquet file or move data file(s) into the `raw/`
folder, which is tracked by LFS.

2. Stage the new files (using `git add`). They should now be tracked by
LFS.
2. Stage the new files (using `git add`). They should now be tracked by LFS.

3. Double-check that Git LFS is configured correctly:
- The output of `git lfs ls-files` should include the new files.
- `git lfs env` should show that `Endpoint` is set to a local path
pointing to the LFS store on GenomeDK.
- `git lfs env` should show that `Endpoint` is set to a local path pointing
to the LFS store on GenomeDK.

4. Commit and push the new data. If you look at the new data file on
GitHub, you should see a pointer file rather than the actual data.
4. Commit and push the new data. If you look at the new data file on GitHub, you
should see a pointer file rather than the actual data.

### Working on a Data Package outside GenomeDK
### Working on a Data Package outside GenomeDK {#outside-genomedk}

If you want to work on a Data Package that uses Git LFS on your own
machine, you can check out the repo without downloading the data files.
In fact, it is not possible to download the data files via Git LFS,
because the LFS store path points to a local folder on GenomeDK that
your machine cannot access.
If you want to work on a Data Package that uses Git LFS on your own machine, you
can check out the repo without downloading the data files. In fact, it is not
possible to download the data files via Git LFS, because the LFS store path
points to a local folder on GenomeDK that your machine cannot access.

You don't need Git LFS installed locally, but you do have to tell Git
not to attempt to fetch data files tracked by Git LFS, as this would
lead to an error. After cloning the repo, run the following from the
root of the Data Package:
You don't need Git LFS installed locally, but you do have to tell Git not to
attempt to fetch data files tracked by Git LFS, as this would lead to an error.
After cloning the repo, run the following from the root of the Data Package:

``` {.bash filename="Terminal"}
```bash {filename="Terminal"}
# Show LFS-tracked files as pointers in the working tree
git config --local filter.lfs.smudge "cat"
# Do not require Git LFS to be installed
Expand All @@ -280,6 +320,6 @@ git config --local filter.lfs.required false
git config --local filter.lfs.process ""
```

With this setup, you won't have access to the actual data or be able to
run scripts that depend on it. However, you can still run other parts of
the code and work on data-dependent scripts without executing them.
With this setup, you won't have access to the actual data or be able to run
scripts that depend on it. However, you can still run other parts of the code
and work on data-dependent scripts without executing them.