Wildland 0.2 - new features, new backends and better performance
We are pleased to announce the release of Wildland Client 0.2.0. This release adds new integrations, brings a number of improvements, and overall performance optimization.
New backends and integrations
With this release we are adding a new SSHFS backend that allows for hosting data on a remote host via SSH. Similarly to the encryption backend, it uses an external FUSE application for the actual work. The backend supports both password-based and key-based authentication. In either case, credentials are stored in the storage manifest (which is encrypted by default).
wl storage create sshfs \ --container mycontainer \ --host <SERVER NAME> \ --ssh-user <LOGIN> \ --pwprompt
The above command will create new storage manifest for
mycontainer container, and prompt for the password.
Wildland Client 0.2 also adds several new intergation backends to allow the user to easily expose and incorporate data from a third party source into their filesystem.
Transpose backend - allows for better integration of different containers into a single forest
GitLab backend - shows issues on GitLab as a container.
wl storage create gitlab \ --container GitLab-p \ --personal-token <ACCESS TOKEN> \ --projectid <PROJECT ID>
An access token can be generated in GitLab’s user settings. The project ID is available at the top of the project overview page.
git backend - gives a read-only access to a git repository.
$ user@wildland-client:~$ wl c create --path /git/demo git-demo Created: /home/user/.config/wildland/containers/git-demo.container.yaml
$ user@wildland-client:~$ wl s create git --url https://gitlab.com/wildland/wildland-client.git --container git-demo Using container: /home/user/.config/wildland/containers/git-demo.container.yaml (/.uuid/8d8df14a-2338-4036-a86b-47ce1ef0be9e) Adding storage 79abb335-307e-4ecf-935b-61070490c59e to container. Saved container /home/user/.config/wildland/containers/git-demo.container.yaml
$ user@wildland-client:~$ wl c mount git-demo Loading containers (from 'git-demo'): 1 Checking container references (from 'git-demo'): 1 Preparing mount of container references (from 'git-demo'): 1 Mounting one storage
$ user@wildland-client:~$ ls wildland/git/demo/ COPYING docker-compose.yml requirements.ci.in wildland-cli Documentation mypy.ini requirements.ci.txt wildland-docker.sh Makefile plugins requirements.dev.in wildland-fuse README.rst pytest.ini requirements.dev.txt wl ci requirements.base.in setup.py docker requirements.base.txt wildland
gitlab-graphql backend utilizes “GraphQL” API to retrieve more information with a single gitlab API request, which makes the process much faster.
Other improvements and optimizations
Cache storage - it is now possible to mount a remote container and cache its contents locally. This will allow for data access even if remote storage disappears or is not reachable. Please note that the initial data download will happen on the first mount, and the process may take an extended period of time (depending on the container’s size).
To use this feature, it is necessary to create a storage template for the cache, usually a local storage in some directory:
$ user@wildland-client:~$ wl template create local cache-local --location /home/user/.cache/wildland
Then, it is possible to mount a container with a cache using the
$ user@wildland-client:~$ wl c mount --cache-template cache-local cat-photos-example
It is now also possible to mount whole forests with a cache (use this feature with care when it comes to big forests!):
$ user@wildland-client:~$ wl f mount --with-cache :/users/pandora:
It may be necessary, however, for larger forests, to raise the max allowed number of inotify instances:
$ sudo sysctl -w fs.inotify.max_user_instances=1024
Sync daemon - background data synchronization is better organized now, with a delegated, separate daemon. It is now possible to handle many more containers, and also to see some information about the progress or issues (like file conflicts). The status is included in the
wl status command. Currently it only shows which containers are synchronized, as well as encountered issues. Future improvements will include a progress indicator.
Pseudomanifest edit - container metadata can now be modified by editing the
.manifest.wildland.yaml file in the mounted container itself. This allows users to use their favorite editor to change some of the properties, instead of having to go to the console. Note, however, that only some fields can be changed this way, i.e. container paths, categories, and the title.
wl status output - now the output lists only the relevant and interesting parts, instead of all (generated) container paths. Most importantly, categories are listed separately, instead of being included in generated paths.
Publisher refactor - the API now allows you to implement publishing into catalogs based not only on files, but also on other kinds of storages. As a side effect, publishing should be faster now.
Several S3 backend optimizations - new changes avoid unnecessary invalidation of the cache. For example, a change into a single file does not throw out the whole cache. An API for bulk writing has also been added. This allows for the publisher to upload manifests, without regenerating the cache several times in the process.
Container (auto-)publishing improvements - containers are not re-published if nothing was changed. Additionally, the check for unpublished containers is moved to the
publish action, and not any container mount.
Several changes were also made to the CLI:
- improved progress reporting (stages that were previously silent now have a progress counter)
- some misleading messages were corrected
- color output was added: errors are printed in red, warnings in yellow
- CLI now handles mount errors individually, instead of interrupting the mount
- syntax for the
wl container modifycommand was simplified, the cumbersome
wl c modify add-path --path ...was replaced with the much more elegant
wl c modify --add-path ...
- added suport for
wl c unmount --save
- most commands do not display sensitive information (e.g. storage credentials) anymore, and show a “Sensitive fields are hidden” message instead. This makes it safer to use the
wlcommand in public. Note, however, that
wl c dumpstill displays the whole manifest.
Docker images were restructured in order to separate several use cases:
- base image - the bare minimum required to run Wildland
- development image
- CI image (for automated tests in gitlab)
- user image - the one that is published into the container registry
- the creation of delegated storage was fixed
- help messages include default option values
- better error reporting
- container removal does not require starting the fuse driver
- Dropbox backend got support for a new token type (the old, long-lived tokens are deprecated by Dropbox)
Wildland Client 0.2.0 is available on our Download page. With the release of the new client we have also updated the Wildland documentation page. Head to docs.wildland.io to read the new tutorials, and learn about the new features and integrations. If you are new to Wildland, please start with A Practical Introduction to Wildland and the Quick Start Guide.