Git Access

CATMA uses a GitLab instance as its backend. All data of your CATMA project is stored in a GitLab project and can be worked with as a Git repository.

CATMA works on a local clone of the Git repository. When you synchronize your CATMA project you actually synchronize the local Git projects with their remote counterparts of the GitLab backend. That means before accessing your data via Gitlab/Git you should always synchronize your CATMA project within CATMA.

Make yourself familiar with Git and Gitlab first!

Overall Structure

A CATMA project is equivalent to a GitLab project.

The CATMA projectID is equivalent to the GitLab project name. The name and the description of your CATMA project are stored in the GitLab project description as a json string.

If you have a CATMA account and want to inspect your CATMA projects in the backend, you can use your credentials to log in to the GitLab backend (your CATMA username and password or your Google account).

User Management

CATMA uses the user, role and permission management of GitLab. If you are working with a team on your CATMA project you will find all participating team members in the Members section of the corresponding GitLab group.

Resources

A CATMA Project usually contains several resources. A resource can either be a Document, an Annotation Collection or a Tagset. All resources are modeled as files which are stored in a Git repository. In the following sections, the setup of the Git repository and the different types of resources are explained.

The Root GitLab Project/Git repository

You will find all your CATMA resources stored in a GitLab project/a Git repository.

The name of the GitLab project corresponds to the internal CATMA project ID (in this example, CATMA_900F812B-69EF-4326-A3E6-58BCFF509719) and the name of the project (in this example, Shakespeare).

So you would find a GitLab project in your account with the name:

CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare

The GitLab project you can see in the GitLab backend is (usually) the latest, synchronized version of the CATMA project. The latest state of the project is stored on the git branch main. When you add a resource to your CATMA Project, it is added to your local GitLab project/Git repository in a separate branch first. When synchronizing in CATMA, your local changes (so new resources, updates to existing resources or deletion of resources) are pushed to the remote server, i.e. the GitLab backend, and merged with the main branch.

This enables synchronization for multiple users – each user has a separate branch, both local and remote, and the latest synchronized state of the project can be found on the main branch on the remote.

The folder structure of a GitLab project/Git repository would look like this for our example project:

CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare
\__ .git
\__ documents\
\__ collections\
\__ tagsets\

Resource Structure

Documents

The Documents are all within the documents folder of your root GitLab/Git project.

Each Document has its own folder which is named after the CATMA ID of the Document.

Each Document folder contains four files:

• header.json – contains metadata of Document
• CATMA_ID_orig.EXT – the original file that was uploaded with a format specific extension
• CATMA_ID.txt – the extracted text in UTF-8 plain/text
• CATMA_ID.json – the indexed types with the start, end and token offset of their tokens, i. e. the word list

The Extracted Text

The file with the extracted text in UTF-8 plain/text (CATMA_ID.txt) serves as the base on which CATMA annotations are built upon. Each annotation has a start and an end offset, all start and end offsets of the Annotations are resolved against the character offsets of the extracted text.

Tagsets

The Tagsets are all stored in the tagsets folder of your GitLab project/Git repository.

Each Tagset has its own folder named after the CATMA ID of the Tagset.

Each Tagset folder contains a header.json file with some meta data like the Tagset’s name, the color etc..

Remember: a tagset contains zero or more Tags that can form a hierarchical structure:

• A Tag has either no parent (top level Tag) or exactly one parent.
• A Tag can have zero or more child Tags.

Each Tag has its own folder named after the Tag’s CATMA ID. Folders for Tags which have a parent Tag are located as subfolders of a folder, named after the parent Tag’s ID.

This sounds more complicated than it is, because in the end each Tag is represented by a file named propertydefs.json. To access all Tags of a Tagset you might want to parse all propertydefs.json files that can be found in the subdirectories of the Tagset’s folder.

The propertydefs.json file contains:

• the name
• the ID, which is a UUID
• the parent ID, which is a UUID but can be empty in case of a top level Tag
• two system properties:
  • the author (catma_markupauthor)
  • the color (catma_displaycolor)

The values of the system properties are to be found in the property named possibleValueList.

The color is encoded as an integer value containing red, green and blue values encoded as bits: red component in bits 16-23, the green component in bits 8-15, and the blue component in bits 0-7. This corresponds to the color encoding in HTML.

Besides the system properties, a Tag can have zero or more user defined Properties. Each Property has a name and a list of possible or proposal values (`possibleValueList) which are presented to the user when applying a Tag.

Collections

The Annotation Collections are all within the collections folder of your GitLab project/Git repository.

Each Collection has its own folder named after the CATMA ID of the Collection.

Each Collection folder contains a header.json file with some meta data like the Collections’s name and the ID of the Document it belongs to (sourceDocumentId).

Annotations

The Annotations are located in a subfolder called annotations. Each Annotation has its own file named after the CATMA ID of the Annotation.

Annotations follow the Web Annotation Data Model and are serialized as JSON-LD.
Each Annotation has a type, i. e. its Tag, and one or more references to possibly non-adjacent (discontinuous markup) text segments. Each Annotation has a timestamp and an author (not to be confused with the author of the Tag). The Annotation inherits the color, the name and the user defined properties from its Tag. A user defined Property of an Annotation can have zero or more values which are either drawn from the set of possible values of the Tag or defined by the user while annotating (ad-hoc values).

Within the Annotation’s file you’ll find the following subsections:

• tagset – the URL of the Tagset GitLab/Git project. This URL also contains the ID of the Tagset.
• tag – the URL of the Tag GitLab/Git project. This URL also contains the ID of the Tag.
• properties – The Properties and their Annotation specific values:
  • system – timestamp and author of the Annotation
  • user – user defined properties with the CATMA ID and a list of values for each property
• target – contains a list of TextPositionSelector selectors with start and end offsets that reference the aforementioned UTF-8 plain/text file with the extracted text

Working with Git

If you want to work with a CATMA project as a Git repository, you can follow this section:

The GitLab backend is accessible at git.catma.de. Once you’ve logged in with your CATMA account credentials (or your Google account) you can access your settings in the upper right corner. On the settings page you will find in the menu on the left the Access Tokens menu item.

Add a Personal Access Token with the api scope enabled.

Make sure you copy the token right after creation and put it somewhere safe. You won’t be able to see the token itself after you leave the page!

Before you start please make sure that your local installation of Git has the right setting for the handling of line endings. The value of core.autocrlf needs to be set to false. You can check the value with:

git config --global --get core.autocrlf

If it doesn’t print anything out then it defaults to false. You can set the value with:

git config --global core.autocrlf false

Note that it is also possible to set this value per Git repository or on the system level. Just make sure that it is set to false for all CATMA Git repositories/projects!

Now you can work with the GitLab API.

For example, to get a list of all of your CATMA Projects:

https://git.catma.de/api/v4/groups/?private_token=THE_TOKEN

or with the parameter GROUPID_OR_NAME set to the CATMA Project ID to get a list of all the resource of a GitLab project/Git repository:

https://git.catma.de/api/v4/groups/GROUPID_OR_NAME/projects/?private_token=THE_TOKEN

Taking the Git URL of the corresponding GitLab/Git project, you can also work with Git to clone a CATMA project locally. Working with the abovementioned Shakespeare CATMA Project, the command to clone the repositor would look like this:

git clone https://git.catma.de/CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare/CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare_root.git

Use the created access token as username and password.

Alternatively, you can also add an SSH key to your account in the GitLab settings and clone a repository with SSH.

git clone git@git-dev.catma.de:username/CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare.git

You can use the cloned repository for backup purposes or to integrate external systems. However, be aware that CATMA can handle only a certain amount of complexity when resolving merge conflicts. You should therefore always resolve conflicts on your side before pushing changes to the remote manually.

Note that you should take care to stick to the aforementioned folder and file structures and formats to avoid errors.