Logo

We are pleased to announce the release of Wildland Client 0.2.0. This release adds new integrations, brings a number of improvements, and overall performance optimization.

New backends and integrations

With this release we are adding a new SSHFS backend that allows for hosting data on a remote host via SSH. Similarly to the encryption backend, it uses an external FUSE application for the actual work. The backend supports both password-based and key-based authentication. In either case, credentials are stored in the storage manifest (which is encrypted by default).

Example usage:

wl storage create sshfs \
     --container mycontainer \
     --host <SERVER NAME> \
     --ssh-user <LOGIN> \
     --pwprompt

The above command will create new storage manifest for mycontainer container, and prompt for the password.

Wildland Client 0.2 also adds several new intergation backends to allow the user to easily expose and incorporate data from a third party source into their filesystem.

Transpose backend - allows for better integration of different containers into a single forest

GitLab backend - shows issues on GitLab as a container.

Example usage:

wl storage create gitlab \
     --container GitLab-p \
     --personal-token <ACCESS TOKEN> \
     --projectid <PROJECT ID>

An access token can be generated in GitLab’s user settings. The project ID is available at the top of the project overview page.

git backend - gives a read-only access to a git repository.

Example usage:

$ user@wildland-client:~$ wl c create --path /git/demo git-demo
Created: /home/user/.config/wildland/containers/git-demo.container.yaml

$ user@wildland-client:~$ wl s create git --url https://gitlab.com/wildland/wildland-client.git --container git-demo 
Using container: /home/user/.config/wildland/containers/git-demo.container.yaml (/.uuid/8d8df14a-2338-4036-a86b-47ce1ef0be9e)
Adding storage 79abb335-307e-4ecf-935b-61070490c59e to container.
Saved container /home/user/.config/wildland/containers/git-demo.container.yaml

$ user@wildland-client:~$ wl c mount git-demo
Loading containers (from 'git-demo'): 1
Checking container references (from 'git-demo'): 1
Preparing mount of container references (from 'git-demo'): 1
Mounting one storage

$ user@wildland-client:~$ ls wildland/git/demo/
COPYING        docker-compose.yml     requirements.ci.in    wildland-cli
Documentation  mypy.ini               requirements.ci.txt   wildland-docker.sh
Makefile       plugins                requirements.dev.in   wildland-fuse
README.rst     pytest.ini             requirements.dev.txt  wl
ci             requirements.base.in   setup.py
docker         requirements.base.txt  wildland

gitlab-graphql backend utilizes “GraphQL” API to retrieve more information with a single gitlab API request, which makes the process much faster.

Other improvements and optimizations

Cache storage - it is now possible to mount a remote container and cache its contents locally. This will allow for data access even if remote storage disappears or is not reachable. Please note that the initial data download will happen on the first mount, and the process may take an extended period of time (depending on the container’s size).

To use this feature, it is necessary to create a storage template for the cache, usually a local storage in some directory:

$ user@wildland-client:~$ wl template create local cache-local --location /home/user/.cache/wildland

Then, it is possible to mount a container with a cache using the --cache-template option:

$ user@wildland-client:~$ wl c mount --cache-template cache-local cat-photos-example

It is now also possible to mount whole forests with a cache (use this feature with care when it comes to big forests!):

$ user@wildland-client:~$ wl f mount --with-cache :/users/pandora:

It may be necessary, however, for larger forests, to raise the max allowed number of inotify instances:

$ sudo sysctl -w fs.inotify.max_user_instances=1024

Sync daemon - background data synchronization is better organized now, with a delegated, separate daemon. It is now possible to handle many more containers, and also to see some information about the progress or issues (like file conflicts). The status is included in the wl status command. Currently it only shows which containers are synchronized, as well as encountered issues. Future improvements will include a progress indicator.

Pseudomanifest edit - container metadata can now be modified by editing the .manifest.wildland.yaml file in the mounted container itself. This allows users to use their favorite editor to change some of the properties, instead of having to go to the console. Note, however, that only some fields can be changed this way, i.e. container paths, categories, and the title.

Improved wl status output - now the output lists only the relevant and interesting parts, instead of all (generated) container paths. Most importantly, categories are listed separately, instead of being included in generated paths.

Publisher refactor - the API now allows you to implement publishing into catalogs based not only on files, but also on other kinds of storages. As a side effect, publishing should be faster now.

Several S3 backend optimizations - new changes avoid unnecessary invalidation of the cache. For example, a change into a single file does not throw out the whole cache. An API for bulk writing has also been added. This allows for the publisher to upload manifests, without regenerating the cache several times in the process.

Container (auto-)publishing improvements - containers are not re-published if nothing was changed. Additionally, the check for unpublished containers is moved to the publish action, and not any container mount.

Several changes were also made to the CLI:

  • improved progress reporting (stages that were previously silent now have a progress counter)
  • some misleading messages were corrected
  • color output was added: errors are printed in red, warnings in yellow
  • CLI now handles mount errors individually, instead of interrupting the mount
  • syntax for the wl container modify command was simplified, the cumbersome wl c modify add-path --path ... was replaced with the much more elegant wl c modify --add-path ...
  • added suport for wl c unmount --save
  • most commands do not display sensitive information (e.g. storage credentials) anymore, and show a “Sensitive fields are hidden” message instead. This makes it safer to use the wl command in public. Note, however, that wl c dump still displays the whole manifest.

Docker images were restructured in order to separate several use cases:

  • base image - the bare minimum required to run Wildland
  • development image
  • CI image (for automated tests in gitlab)
  • user image - the one that is published into the container registry

Other fixes:

  • the creation of delegated storage was fixed
  • help messages include default option values
  • better error reporting
  • container removal does not require starting the fuse driver
  • Dropbox backend got support for a new token type (the old, long-lived tokens are deprecated by Dropbox)

Download

Wildland Client 0.2.0 is available on our Download page. With the release of the new client we have also updated the Wildland documentation page. Head to docs.wildland.io to read the new tutorials, and learn about the new features and integrations. If you are new to Wildland, please start with A Practical Introduction to Wildland and the Quick Start Guide.