On September 25, 2020, the Golem Foundation had its first live event – the WildCon1, an internal (but open to the public) seminar on Wildland.
Wildland is a highly ambitious project to decentralize the web, help individuals to take back control of their data, and free users from dependency on on-line service providers.
Around 25 people gathered at the MindSpace in Warsaw, Poland to take part in the event. There were high-level overviews of the project, demos of Wildland’s early implementations, and some on-the-fly bug fixing. Everything was caught live on camera, and the videos from the seminar will be soon made available for the general public. For now, here is a detailed written report from the whole event.
1. Wildland’s high-level rationale
When taking on a big and complex project, it’s a good idea to ask yourself why you’re doing it, and what you are trying to achieve with it. Wildland is Golem Foundation’s response to the problems of modern IT technology.
As Julian Zawistowski explained during one of the early Wildcon1 presentations, “the development of consumer IT technology took a wrong path over the last decade. Most people rarely use file systems anymore. They do not interact with raw data.” Instead, they have become more and more dependent on various on-line services, which are provided to them by private, centralized entities whose goals do not align with users’ interests.
These services may be convenient to the user, and quite often are free to use, but they come with some heavy strings attached. The loss of privacy inherent in online services is a well-known issue. So is the monetization of users’ online behavior. What is however not acknowledged often enough is that service providers have an almost unlimited power to deny access to their services, for whatever reason they choose to include in their policies.
Thus, users of modern on-line services have lost their privacy, they have only indirect access to most of their data, and can be cut off services without a warning. Wildland aims to change all of that. “Our ultimate goal is to create a future in which the user, and not a service provider, is a digital sovereign”.
To achieve this goal Wildland completely redesigns data management, by switching from a service-oriented to the data-oriented paradigm, and giving individuals explicit control over the choice of data storage infrastructure.
2. Self-defined containers – Wildland’s building blocks
Wildland uses a three-layer architecture. The top-most layer is the information-layer, where the user has access to all her files. For most of the users, this will be the only Wildland layer, with which they will interact directly. Self-defined data containers provide the building blocks for this layer. As Joanna Rutkowska explained during one of her presentations, “Wildland containers are similar to Docker containers, except that dockers are for code, and Wildland containers can store any type of information”. What makes Wildland containers truly unique, however, is that they are self-defined.
Every Wildland container has a manifest attached to it. The manifest defines the owner of the container, a list of its contents, various access rights, a backend on which the container is stored, and the various paths through which the container can be accessed. This last feature is worth stressing.
Unlike most files systems Wildland does not restrict users to only one access-path to their information. To the contrary, it encourages the user to define as many paths to her data, as she finds useful. This makes paths in Wildland similar to tags and lets users organize their knowledge in multiple ways. For example, the container which contains the epub version of the “Anarchy, State and Utopia” by Robert Nozick (the book referred to by Joanna in one of her presentations, which given the nature of Wildland was probably not a random choice) could be addressed by a user with the following paths:
The uniqueness of Wildland’s approach to addressing lies in the fact that all those paths are equally valid. A Wildland client will make the container available in all the different paths that are specified in the containers’ manifest. This approach allows for the level of multi-categorization that is not available in any other file management system.
Wildland containers are not purely theoretical concepts. A directory can be easily turned into a Wildland container by adding the appropriate Wildland manifest to it. This has been demonstrated live by Marek Marczykowski-Górecki and Paweł Marczewski from the famous Invisible Things Lab, the team that is mostly known for their work on Qubes OS.
During their presentation, Marek and Paweł created a Wildland container containing some cute cat photos, mounted it to an existing directory structure, and navigated to it using different paths listed in the manifest. They also demonstrated that a Wildland container can be hosted in parallel on a local hard drive and some cloud storage bucket. For each backend, different access policies were defined (read and write, and read-only respectively). The container hosted in a cloud storage service even included an index file, thanks to which it could be browsed like a website.
Put together, the user’s containers form what Golem Foundation calls, an individual “Forest of Knowledge” (the choice of this “organic” name for something that is built out of “containers”, will be made clear later in the article). This “forest” corresponds to a non-hierarchical namespace. Browsing this namespace does not require the use of any specific user interface. As Marek and Paweł demonstrated during the seminar, users can navigate through their forests using Finder, Nautilus, Dolphin, Midnight Commander, a terminal, or any other file manager.
As Joanna explains, “with Wildland I can use multiple categories to organize my knowledge, and I can navigate through this complex forest of information using a simple file management tool. This allows for interesting use cases. Using a relatively simple Python script I can have my mailbox exposed as N-numbers of containers. Each container represents a different email message, and each container has categories like time, actors (senders), topic, etc. This action would make my mailbox available as a system of files that I could then navigate through using the Finder. And the same goes for calendar events. In essence, all the data sources which we are now consuming through different applications could be represented as Wildland containers, and all those containers could be integrated into a file system.”
To show that this would, indeed, be possible, Wojtek Porczyk from the Invisible Things Lab demoed a backend that plugged into a popular iOS/macOS note-taking app Bear, and translated all the notes written in it into Wildland containers.
In Wildland individual forests are identified by UIDs (pubkeys), and users can have multiple forests, each identified by a different pubkey. A user can set up a different forest for every device she owns – one for her laptop, another for her desktop, yet another one for her tablet – or for different use cases: work, private, etc. This allows for compartmentalization of data that many users should find very useful.
3. Bottom-up directories
Wildland also allows users to share part of their forest (i.e. specific containers) with other users.
Making containers residing in remote forests available to other users required the Golem Foundation’s team to come up with some sort of a lookup system. This is a very challenging problem.
Most systems that facilitate resource addressing in vast networks rely on centralization. This is most obvious in the case of the Domain Name System, which ultimately relies on a small number of root name servers. This approach is completely alien to the Golem Foundation team. “We do not want any centralization in our system. Even the Ethereum Name Service is too centralized for us”, Joanna claims. Instead of relying on a centralized hierarchical database or a set of smart contracts, Wildland implements a novel notion of bottom-up directories.
Wildlands lookup system uses “cascading addressing” and “Bridge Manifests”. A Bridge manifest is a container that stores information about another user’s ID. Specifically, it contains her pubkey. When a Wildland client encounters such a container on its path, it realizes that it needs to “jump” to a forest belonging to another user.
Take as an example the following Wildland address:
Wildland: is a protocol identifier, similar to HTTPS, FTP, etc.
pubkey1 identifies a forest belonging to a specific user.
pubkey1:path1 points to a specific container in the
pubkey1 user’s forest. This container contains a pubkey that belongs to another user and identifies her forest (
pubkey1:path1 container is an example of a Bridge Manifest. It informs the agent who parses the address, that it needs to jump to a forest belonging to the
pubkey2 user. Thanks to the information contained in this Bridge Manifest, a Wildland agent knows that
path2 in our example points to a container that resides in the
pubkey2 user’s forest.
This process of cascading addressing allows for the creation of bottom-up directories. After all, the
pubkey1 user can be seen as a “directory”, which lets others find the
pubkey2 user. The most interesting feature of this lookup system is that it does not rely on any centralized logic. There may be many different directories, established by various users and organizations. The Wildland lookup system is built on a bottom-up principle. This should explain why the Golem Foundation team is so fond of the names relating to naturally-growing phenomena such as forests and other forms of wildlife.
The process of sharing containers between different Wildland users was demonstrated live by Marek Marczykowski-Górecki from the Invisible Things Lab. During Marek’s demonstration every step that is needed to share the container (like e.g. the creation of bridge manifests) was done manually, to illustrate the underlying logic of bottom-up directories, but as he assured the public, “in Wildland’s first publicly available implementation all this sharing and discovering of resources should be automated”.
4. Backend agnosticism
Organizing knowledge is one thing. Storing data that provides the basis for our knowledge is another thing altogether, and it comes with its own set of problems. As Joanna explains, “We see infrastructure as a necessary evil. In the ideal, Platonic world, we would only have information.” Unfortunately, as it is right now, we have to store the information somewhere, which in our digital world means putting our files on some hard drive. Hence, below the Forest of Knowledge rests Wildland’s infrastructure layer.
The infrastructure layer is where the users’ containers are stored. Wildland does not rely on one specific backend. The containers can be stored in the user’s local file system, in network-attached storage, or some cloud storage service. Wildland is designed in such a way as to make the containers easily movable between different infrastructure. By keeping all the meta-information about the container into the container itself, Wildland effectively “unsticks” the container from infrastructure.
Thanks to this approach, “today my container can be stored on Amazon S3, next month on some Google Cloud server, and later somewhere else. The fact that containers are self-defined means that there is no need to update the address contained in the manifest”, Joanna explains.
5. Backend leasing made easy through tokenomics
All this can be true only if the containers can be moved seamlessly between different backends. Today this is not the case. To move her files from one cloud storage to another, a user needs to have accounts with two separate cloud providers. Setting them up is not always easy, quite often the action requires familiarizing yourself with the relevant technical documentation, and it usually involves a payment that is processed by a third party.
This problem is solved in Wildland by its lowest layer, the marketplace. The marketplace is where the contracts for storage (and in the future data processing, see below) take place. It’s here that the user acquires credentials necessary to use infrastructure offered by providers. One of the many unique features of the Golem Foundation’s project is that in Wildland this transaction will be automated.
“The beauty of our system is that you do not have to create the storage manifest manually. There is no need for you to set up an Amazon S3 account, log into your AWS management console, or to read the user’s manual. In Wildland you get your credentials on a market, using automated agents and smart contracts. In fact, in our marketplace there is no direct communication between a user and a provider”, Julian claims.
The current implementation of Wildland’s transaction system was demoed during WildCon1 by one of the Golem Foundation’s developers Paweł “Pepesza” Peregud.
Transactions in Wildland are conducted using an Ethereum based smart-contract (the so-called Unified Payment System, or UPS for short). There are two parties to such a contract: a user who is looking for storage space and a provider. Users do not interact directly with smart contracts, they are using automated agents working on their behalf instead.
Every provider registers himself in a smart contract. His offer is then cryptographically signed and listed among the offers of other providers in a publicly accessible Wildland container.
To find suitable storage for the user, the agent matches the user’s requirements specified in her manifest with offers listed in the service aggregator container. Once the agent finds an offer matching the needs of the user, it submits a transaction to the UPS smart contract with the appropriate payment. If the provider accepts the transaction, he collects the payment and publishes the encrypted credentials on a blockchain. Those credentials are then used by the user to access the backend.
All this may seem complicated at first glance, but the Golem Foundation team is working on automating the whole process. Users will only need to top up their wallets and specify their needs regarding the storage in a manifest. Everything else will be done seamlessly by a Wildland agent. As Julian explains, “the adoption of cryptocurrencies among the general public is still low, and we do not want users to interact with smart contracts. We aim to make the transaction as simple as possible. All we want from a user is that she has a wallet (and a backup, just in case) and that she specifies what she wants from a provider – at minimum the amount of storage space she needs, and the maximum price she is willing to pay for it. A Wildland agent working on her behalf will find suitable storage for her, pay for it, and acquire all the necessary credentials via an Ethereum smart contract”.
For the Wildland market to be efficient there has to be a large number of providers on it. The integration of various backends with Wildland will require various approaches, but Golem Foundation believes that overall this process should not be particularly complicated. There was, however, one issue that required the team’s special attention.
Amazon Web Services – a dominant firm in the cloud marketplace – has no built-in metering mechanism. Without such a mechanism providers who have bought storage space from AWS will have trouble resealing it to end-users. To resolve this problem the Golem Foundation team created an S3 proxy backend, which was shown live during a demo presentation by Mateusz Koczan.
This proxy hands out to individual users the credentials they have acquired, using the aforementioned agent, and then it monitors their behavior, to make sure they are not exceeding their quota. During the live demo, Koczan tried to upload some files into his S3 bucket, but since the files were bigger than the storage space he had acquired from Amazon Web Services, his action was blocked by the proxy. “Without the proxy, the files would be uploaded, the quota would be exceeded, and I would be charged with an additional fee”.
As Joanna comments, “this proxy backend can be used by anyone who wants to be a provider in our network and uses mass storage that doesn’t have reselling or metering capabilities built-in. We believe this is a much better mechanism of usage control than the one provided by AWS itself.”
5. Hooks – bringing automation to the people
In the current Wildland’s implementation a container is like a directory that is stored somewhere. In the next iteration of the project, the Golem Foundation would like to add “hooks” to Wildland containers.
Think about the hooks as lambda functions – simple instructions performing small tasks for the user. Imagine a small Python script that gathers headlines from your favorite websites through https, downloads them onto your hard drive, and generates a pdf file that then rests in a container. With a small hook, you can turn your container into your press briefing agent.
As Joanna acknowledges, all this can be done today using a Docker container. “But to accomplish this simple task you need to set up an account on some cloud storage services, you need to have some DevOps skills, etc. In our model, all you need is a short Python script. We do not expect users to write their own scripts, of course. We want them to use scripts they have found on GitHub. Or even better, that they have downloaded from something akin to a “hook store”. What matters is that the script they have acquired will be running inside a container that is being controlled by the users themselves. In this way, we will bring automation to regular people. Automation that is controlled by the people.”
6. Backends stacking
Every Wildland container needs a backend. During one of the demos presented at Wildcon1 Marek and Paweł from the Invisible Things Labs showed that a container can have multiple backends running in parallel (for example, one local with “read & write” access and another public with read-only access). As of today, the replication of content between various backends needs to be done manually, but in the version that will be released to the public, this process will be automated.
Backends in Wildland can work not only in parallel but on top of each other. This process is known as backend stacking and allows for interesting applications. You can put a processing backend on top of a simple file-storing backend. This processing can take many forms, for example, encryption or integrity protection. “With this technology, someone can write a plugin for on-the-fly encryption, and turn any backend into end-to-end encrypted storage”, Joanna says.
During Wildcon1 Marek Marczykowski-Górecki from the Invisible Things Lab presented a simpler demonstration of a backend stacking. On the bottom of his stack, he put a container with a couple of cute cat photos. Then he stacked another container on top of it. The top backend container extracted modification time from the cat photos metadata and then sorted them accordingly in another (third) container.
7. Wildland’s multi-user use cases
Most of the WildCon1 presentations focused on single-user use cases, but as Joanna explained during her final talk, Wildland architecture also allows for the creation of decentralized and user-controlled social networking apps. She based her example on Facebook Events – a Facebook feature that lets users find out what’s going on in their communities and keep track of who’s going where. The feature as it is implemented on Facebook has many obvious problems relating to privacy, monetization, denial of service, etc. In Wildland the same functionality will be possible to achieve without relying on a centralized server or even one common infrastructure. Wildland users can expose their containers to other users via different backends – IPFS, S3, Google Cloud, and so on.
“Users will be able to sign up for a forest set up by some social club, for example, Jazz Lovers Warsaw. The trees in this forest would represent individual users who signed up for this club. The club will list the events that made be of interest to the members of this group in a container that is accessible to all of them. We can have a simple Python script that traverses this forest and looks for which of the events listed in the container were bookmarked by various club members. In Wildland it is a simple operation, and what is important is that this whole algorithm rests in the hands of a user. No centralized entity is needed.”
8. Wildland and economic sustainability
Wildland has many original features that many users may find useful. The question is can this novel ecosystem be nurtured sustainably? This issue has been tackled during Julian’s last presentation. As he noted, decentralized projects face severe governance issues. Two problems are especially important here: who should have the right to decide on questions related to the organization, and how to encourage someone who has this right to exercise it in practice.
Wildland will try to solve these problems with market measures – a combination of a previously outlined Unified Payment System, and the so-called User-Defined Organization.
Three key actors will interact with each other in Wildland: users, providers, and builders (developers who work on projects that make Wildland more attractive to users and providers, like e.g. apps or hooks).
As previously noted, all transactions that take place in Wildland will be conducted with the use of a smart contract. In practice, this whole process would work like this.
Wildland will use some kind of a stablecoin for payments. Every payment for the usage of services (like file hosting or data processing) will be divided into three parts. Most of this payment will go to the provider. The rest of it will be divided into two parts.
The first of the remaining parts will be converted using Uniswap into Golem Network Token (GNT) with a 1:1 exchange ratio. The GNT will be then permanently removed from the system and the corresponding number of non-tradable Proof of Usage tokens will be generated. These tokens will be then allocated to the party that initialized the transaction. The rest of the payment will form the so-called “build fee” and will be allocated into a “build fund”.
The Proof of Usage will be like a decision right in an organization. The more a person will use Wildland services the more Proof of Usage tokens she will be rewarded with, and the more decision power she will gain.
This power will allow the holders of the Proof of Usage tokens to decide on how the funds collected from the build fees are to be spent and participate in Wildland’s User-Defined Organization which among several other things determines the parameters of the Proof of Usage token generation process.
The exact split of the payment has not been decided yet, but as Julian said during the presentation, the Proof of Usage fee and the build fee combined should not exceed 10% of the whole payment.
This complex process will ensure that Wildland’s heaviest users will have the strongest influence on its future development. “We believe that our approach to governance is going to generate some interesting results. Imagine, for example, a social media platform that has been developed to suit the needs of its users and not the wishes of the shareholders. It would probably look rather different from what we are used to right now”.