Git Access

Overview

Every CATMA project is a Git repository hosted on our GitLab instance at git.catma.de. This page explains how that repository is laid out and how to clone and read it with standard Git and GitLab tooling. It applies to CATMA 7; if you have code written against CATMA 6 data, see the migration appendix at the end. CATMA-specific terms are explained in the glossary.

Before you start

1. Have a CATMA account. Your CATMA credentials (username and password, or Google sign-in) also log you in at git.catma.de.
2. Synchronize first. CATMA works on a local clone of your project. The GitLab repository only reflects the state you last synchronized from within CATMA. Always synchronize before reading data via Git or the GitLab API.
3. Know Git and GitLab. This page assumes working knowledge of Git, the GitLab web interface and the GitLab REST API.

How a CATMA project maps to GitLab

One CATMA project is exactly one GitLab project, i.e. one Git repository. Here you can see the project-specific information and resources which are visible in CATMA and their counterpart on GitLab:

All examples on this page use one project: CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare, owned by user alice with a second member bob. Its Git URL is https://git.catma.de/alice/CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare.git. It contains one document, Hamlet, one annotation collection, Hamlet Default Annotations, and one tagset, Dramatic Structure, with a tag Soliloquy.

Branches

The repository has one integrated branch plus one branch per project member:

• master – the integrated project state. It only moves when a member synchronizes from within CATMA, so it can lag behind the members’ own work.
• <username> – one branch per member, named after their CATMA username (alice and bob in the example). This is where that member’s local changes are pushed. A user branch can be well ahead of master.

Which branch to read depends on what you want: master for the agreed, merged, and therefore synchronized state of the whole team (what CATMA shows in the Synchronized view mode); a user branch for one person’s latest work, including anything they have not yet synchronized.

Repository layout

All three resource types are plain files under three top-level folders. Every resource has its own subfolder named after its CATMA ID, and every resource folder starts with a header.json holding its metadata.

CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare/
├── .git/
├── documents/
│   └── <documentId>/
│       ├── header.json
│       ├── <documentId>_orig.<type>
│       ├── <documentId>.txt
│       └── <documentId>.json
├── tagsets/
│   └── <tagsetId>/
│       ├── header.json
│       └── <tagId>/
│           ├── propertydefs.json
│           └── <childTagId>/
│               └── propertydefs.json
└── collections/
    └── <collectionId>/
        ├── header.json
        └── annotations/
            ├── alice_0.json
            ├── alice_1.json
            └── bob_0.json

CATMA IDs are upper-case UUIDs with a prefix that marks the resource type: D_ for documents, C_ for collections, T_ for tagsets and CATMA_ for tags, properties and annotations. In the example project the Hamlet document is D_7C2E4A91-3B5F-4D8A-9E6C-0F1B2D3A4C5E, its collection Hamlet Default Annotations is C_2F0B7C41-5E6D-4A3B-9C8E-1D2F3A4B5C6D and the tagset Dramatic Structure is T_6E3A9B2C-1F4D-4C7E-8A5B-0D9E2F1A3B4C. IDs are stable across renames and synchronizations. The following three sections describe each folder in turn.

Documents

Each document folder under documents/ (for Hamlet: documents/D_7C2E4A91-3B5F-4D8A-9E6C-0F1B2D3A4C5E/) holds four files:

File	Contents
header.json	Document metadata in three groups: content (title, author, publisher, description), indexing (locale, tokenizer settings) and technical (charset, file type, checksum, responsible user); see below
<documentId>_orig.<type>	The file exactly as uploaded. The suffix is not the original extension but the file type according to the upload’s MIME type: _orig.text for text/plain and text/html, _orig.application for application/pdf, DOCX and other binary formats, _orig.unknown if no MIME type was detected. The original file name and full MIME type are in header.json
<documentId>.txt	The extracted text as UTF-8 plain text
<documentId>.json	The term index: every token type with the start offset, end offset and token position of each occurrence

The extracted text is the reference for all offsets

Annotations never point into the original file. Every annotation stores start and end character offsets into <documentId>.txt. To resolve an annotation to its text, read that file as UTF-8 and slice it at those offsets.

Two consequences follow:

• Do not modify <documentId>.txt. Even a changed line ending shifts every offset after it and silently corrupts all annotations on the document.
• When you clone the repository, make sure Git does not convert line endings (see Working with the repository below).

Tagsets and tags

Each tagset folder under tagsets/ (for Dramatic Structure: tagsets/T_6E3A9B2C-1F4D-4C7E-8A5B-0D9E2F1A3B4C/) holds a header.json (name, description, ID) and one subfolder per tag. Tag folders nest to mirror the tag hierarchy: a top-level tag’s folder sits directly in the tagset folder, a child tag’s folder sits inside its parent’s folder. A tag has at most one parent and any number of children.

The tag itself is a single file, propertydefs.json. To load every tag of a tagset, walk the tagset folder recursively and parse every propertydefs.json you find; the folder nesting is redundant with the parentUuid field, so you can rebuild the hierarchy either way.

propertydefs.json

Field	Meaning
name	Tag name as shown in CATMA
uuid	The tag’s CATMA ID
parentUuid	ID of the parent tag; empty string for a top-level tag
tagsetDefinitionUuid	ID of the tagset the tag belongs to
systemPropertyDefinitions	The three built-in properties, keyed by their CATMA ID. Each entry is an object with name, uuid and possibleValueList; the actual value is the single element of possibleValueList. The IDs are derived from the property name and therefore identical in every project
catma_markupauthor	Username of the member who created the tag.
catma_displaycolor	The tag’s colour as a signed 32-bit integer in Java Color.getRGB() layout: alpha in bits 24–31, red 16–23, green 8–15, blue 0–7. Mask with 0xFFFFFF to get the CSS hex value, e.g. -408455 → #F9C479.
catma_markuptimestamp	Always present with an empty possibleValueList. The definition itself holds no value: the actual timestamp is stored on each annotation, in body.properties.system, keyed by this property’s ID.
userDefinedPropertyDefinitions	Properties the project team defined on this tag, keyed by their CATMA ID; {} if there are none

{
  "name": "Soliloquy",
  "parentUuid": "",
  "systemPropertyDefinitions": {
    "CATMA_54A5F93F-5333-3F0D-92F7-7BD5930DB9E6": {
      "name": "catma_markuptimestamp",
      "possibleValueList": [],
      "uuid": "CATMA_54A5F93F-5333-3F0D-92F7-7BD5930DB9E6"
    },
    "CATMA_A309D8FB-C5B8-33C9-A1B1-8CF7A5548C6A": {
      "name": "catma_displaycolor",
      "possibleValueList": ["-408455"],
      "uuid": "CATMA_A309D8FB-C5B8-33C9-A1B1-8CF7A5548C6A"
    },
    "CATMA_AB27F1D4-303A-3622-BB2C-72C310D0C1BF": {
      "name": "catma_markupauthor",
      "possibleValueList": ["alice"],
      "uuid": "CATMA_AB27F1D4-303A-3622-BB2C-72C310D0C1BF"
    }
  },
  "tagsetDefinitionUuid": "T_6E3A9B2C-1F4D-4C7E-8A5B-0D9E2F1A3B4C",
  "userDefinedPropertyDefinitions": {},
  "uuid": "CATMA_3B7D1E5F-9A2C-4F8E-B6D0-1C4A7E9F2B3D"
}

User-defined property definitions have the same shape as the system ones (name, uuid, possibleValueList). For them, possibleValueList holds the proposed property values offered when the tag is applied; annotators may still enter ad-hoc values.

Annotation collections and annotations

Each annotation collection folder under collections/ (for Hamlet Default Annotations: collections/C_2F0B7C41-5E6D-4A3B-9C8E-1D2F3A4B5C6D/) holds a header.json and an annotations/ subfolder. The header carries name, description, author, publisher, responsibleUser (the CATMA username of the member responsible for the collection), forkedFromCommitURL (set only on collections that were copied) and sourceDocumentId, the ID of the one document the collection belongs to:

{
  "author": null,
  "description": null,
  "forkedFromCommitURL": null,
  "name": "Hamlet Default Annotations",
  "publisher": null,
  "responsibleUser": "alice",
  "sourceDocumentId": "D_7C2E4A91-3B5F-4D8A-9E6C-0F1B2D3A4C5E"
}

Page files, not one file per annotation

Annotations are stored in page files named <username>_<pageNo>.json, one series per project member who has annotated in the collection (alice_0.json, alice_1.json, bob_0.json in the example). Each page file is a JSON array of annotation objects. To load all annotations of a collection, read every *.json file in annotations/ and concatenate the arrays; the file names carry no information you need beyond that.

Code written for CATMA 6, which expected one object per file named <annotationId>.json, will fail on CATMA 7 data.

Annotation object

Each annotation follows the W3C Web Annotation Data Model, serialized as JSON-LD.

Key	Contents
@context	JSON-LD context of the Web Annotation vocabulary
type	Always Annotation
id	The annotation’s CATMA ID
body.tagset	Reference to the tagset; the string ends in the tagset ID
body.tag	Reference to the tag; the string ends in the tag ID
body.properties.system	catma_markupauthor: who made the annotation (not who made the tag)
body.properties.user	User-defined properties, keyed by property ID, each with a list of values; values come from the tag’s possibleValueList or were typed ad hoc
target	One or more TextPositionSelector items, each with start and end character offsets into <documentId>.txt. More than one selector means a discontinuous annotation

The annotation inherits its name and color from the tag at display time; neither is stored on the annotation.

Working with the repository

You do not need Git to look at a project: after logging in at git.catma.de you can browse every file in the web interface and download single files or the whole repository as a ZIP archive (Code → Download source code). The steps below are for working with a local clone.

1. Create an access token

1. Log in at git.catma.de with your CATMA credentials.
2. Open your avatar (top right) → Preferences → Access Tokens.
3. Create a personal access token with the api scope.
4. Copy the token immediately and store it safely. GitLab shows it only once.

Note that all personal access tokens have an expiration date, which you can set when you create them. For read-only access to a project, you can also use the access token from the app.catma.de interface (account menu → Get Access Token). It is valid for up to three months and gives read-only access to all projects you own or have joined.

2. Disable line-ending conversion

CATMA resolves annotation offsets against the bytes of <documentId>.txt. If Git rewrites line endings on checkout, every offset after the first newline is wrong. core.autocrlf must be false for every CATMA clone.

Check the current value:

git config --get core.autocrlf

No output means it is already false. Otherwise either set it for all repositories on your machine:

git config --global core.autocrlf false

or leave your global setting alone and pass -c core.autocrlf=false to git clone as shown in step 4, which stores the setting in that one repository before any file is checked out. Setting it inside an existing clone is too late: files already checked out keep their converted line endings.

3. Find your projects via the GitLab UI

Log in at git.catma.de and open your project list. It shows every CATMA project you are a member of, named with the CATMA project ID plus the project name. Open a project and copy its HTTPS clone URL from Code.

Side note: you can also list your projects through the GitLab API, using an API call to /api/v4/projects?membership=true on git.catma.de. Each entry carries the project’s name, description (JSON with the CATMA name and description) and http_url_to_repo.

4. Clone

Over HTTPS, using the token as password (any username works):

git clone -c core.autocrlf=false https://oauth2:THE_TOKEN@git.catma.de/alice/CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare.git

Or let Git prompt for credentials and enter the token as the password:

git clone -c core.autocrlf=false https://git.catma.de/alice/CATMA_900F812B-69EF-4326-A3E6-58BCFF509719_Shakespeare.git

alice is the username of the project owner, not necessarily yours; take the URL from GitLab rather than constructing it. To refresh a clone later, run git pull.

5. Pick a branch

A fresh clone checks out master, the integrated state. To read one member’s latest, possibly un-integrated work, check out their branch, e.g. bob‘s:

git checkout bob

List the available branches with git fetch --all && git branch -r.

6. Putting it together: reading one annotation

With the clone in place, this is how the files relate when you want to see what a given annotation says about the related text:

1. Open collections/<collectionId>/header.json and note sourceDocumentId (for Hamlet Default Annotations: D_7C2E4A91-3B5F-4D8A-9E6C-0F1B2D3A4C5E).
2. Read the extracted text documents/<sourceDocumentId>/<sourceDocumentId>.txt as UTF-8.
3. Read every collections/<collectionId>/annotations/*.json and concatenate the arrays; pick an annotation.
4. For each TextPositionSelector in its target, take the characters from start to end of the text from step 2 – that is the annotated passage.
5. Take the tag ID from the end of body.tag, find the folder with that name under tagsets/, and read its propertydefs.json for the tag’s name, colour and property definitions. The annotation’s own property values are in body.properties.user, keyed by the same property IDs.

To get a document’s title, author or original file name, read documents/<documentId>/header.json.

Writing changes back

A clone is safe to use for backups, archiving and feeding external systems. Pushing changes back is possible but at your own risk: CATMA’s capability to handle merge conflicts only extends to simple cases which can be resolved automatically, and malformed files will make a project unloadable. If you do push:

• Keep exactly the folder, file-name and JSON structures described above.
• Never change <documentId>.txt, <documentId>_orig.<type> or <documentID>.json, since the markers for the start and end of an annotation rely upon a consistent source document.
• Push to your own user branch, not to master, and resolve any conflicts locally before pushing.
• Synchronize in CATMA afterwards and check that the project still opens.

If you’re stuck or run into unexpected problem when following this guide, you can contact the CATMA support.

Appendix: Migrating from CATMA 6

If you have scripts written against CATMA 6 repositories, a few things changed:

CATMA 6	CATMA 7
A project was a GitLab group; each resource was its own GitLab project, tied together as submodules of a _root repository. Clone with --recurse-submodules.	A project is one GitLab project and one repository. Clone normally; update with git pull.
Clone URL: https://git.catma.de/<projectId>_<name>/<projectId>_<name>_root.git	Clone URL: https://git.catma.de/<ownerUsername>/<projectId>_<name>.git
List projects via /api/v4/groups/	List projects via /api/v4/projects/
One annotation per file, annotations/<annotationId>.json, containing one JSON object	Page files annotations/<username>_<pageNo>.json, each containing a JSON array of annotation objects
Only a master branch	master plus one branch per member, named after their username; master may lag behind user branches

The folder layout, header.json files, propertydefs.json and the annotation object format are unchanged.