This document is the first introduction for ValOS technicians - the primary infrastructure developers and operators - to the ValOS ecosystem and its infrastructure toolchains and workflows.

This document is part of the vault workspace @valos/kernel (of domain @valos/kernel) which has the description: `ValOS common infrastructure tools and libraries monorepository`.

NOTE(2019-08): This document is severely `OUTDATED and SEMANTICALLY UNRELIABLE`. This content is what used to be in DEVOPS.md, is relatively old and needs to be revised. The specification needs to be extracted to be a separate document from the devops workflows.

§ How do I enable valonauts and expand the Valospace?

As a technician you develop and operate the ValOS fabric, the globally distributed web-like infrastructure of servers, services and components which underlies the Valospace.

You use vlm and employ your existing, likely professional knowledge of JavaScript, Node.js, DevOps, backend, library and other software development skills.

You create new open source and/or proprietary node packages. These permanently create new fabric functionalities and expand the Valospace by integrating it to old world systems.

§ Technician role sub-profiles

Technician sub-profiles detail the typical stages of software development and operations workflows within ValOS ecosystem. A technician should read the detail docs of their own profiles but also understand the at least the abstract principles of all other profiles, which are expanded below.

§ How developers set up their development environments

§ How contributors produce git PR's and new fabric packages

§ How testers assure quality before releases

§ How administrators manage infrastructure resources

§ How DevOps workflows are implemented using valma

§ How hackers have fun whilst avoiding pain

§ How to add new profiles

Create a PR in github against @valos/kernel/revdocs/technician.revdoc.js

§ *IMPORTANT

This document, like most of the more principled and less pragmatic documents must be understood as strong speculation and as an architectural exercise (although a very serious one) on what-could-be.

This rings especially true for the primary focus of this document, ie. for ValOS Core as the fully functional open source core around which the often proprietary ValOS ecosystem would expand. This is an ambitious, even presumptuous attempt to facilitate the fruitful side-by-side living of open source communities as well as private enterprises as a combination of technical and social architecture.

(The technical part is the ambitious part and the social part is the presumptuous part. :D)

§ ValOS specification

This document has two purposes. Firstly it provides the initial specification of Valaa Open System (ValOS) and secondly it serves as the introduction for Development Operations.

ValOS specification is provided as quoted and numbered ValOS rules.
valos-vault-1.1: ValOS packages are all the npmjs.com packages with @valos scope which don't have cyclic dependencies with other @valos scope packages.
valos-vault-1.2: Valaa Open System (ValOS is defined to be the contents of all ValOS packages and nothing else.
valos-vault-1.3: ValOS specification consists of all files in all ValOS packages whose path in the package matches the JS regex /^specifications\/\w*.md$/ and nothing else.
valos-vault-1.4: Rules in a package are considered to be more specific than the rules in its package dependency tree. More specific rules take precedence.
valos-vault-1.5: All packages which conform to ValOS specification are called ValOS packages. These packages are inclusively considered part of the ValOS ecosystem.

The system is structured into purpose-oriented vertical domains and into four content oriented horizontal utility layers.

Each utility layer is named by the content or service is provides and builds or depends on the previous layers.
  • files layer provides files via git repositories.
  • packages layer provides npm packages via npm registry. These packages are created by git repositories called vaults.

  • authorities layer provides the ValOS live authority service APIs. Authorities are deployed and managed with opspace workspaces.

  • chronicles layer provides the resource content as chronicle events and bvobs. These are served via authority service APIs.

TODO(iridian): Figure out and describe the concrete role of domains. NOTE(iridian, 2019-08): Domains are now mostly figured out; they are administrative, organisational units and namespaces.

A DevOp manages these layers by scripts that are delivered inside the packages alongside their main content. A command line tool called ValOS Medium or vlm is used to discover and invoke these scripts as valma commands.

From a DevOps perspective valma, vault and opspace are the three concrete core mechanisms that everything else ties into.

Valma and specific domains are specified in other documents and as such are only briefly described here.

The utility layers are common to everything form the bulk of this document. They are fully specified at the last part of this document after all the brief descriptions.

§ ValOS `domains` are cross-stack slices, each with a well-defined purpose

valos-vault-2.1: A domain is the collection of all the systems, interactions and dynamics which exclusively serve a particular purpose.
valos-vault-2.2: A domain must define the purpose, describe its producers and consumers and should provide a direction for technical and operational structure as well.
For example the @valos/kernel domain provides the essential central ValOS code for developing new ValOS infrastructure software. Its main producers are the kernel software developers and its consumers are the infrastructure software developers. It revolves around developing code as library packages.

§ `valma` - a convenience CLI to context-dependent command scripts

valma (vlm in CLI) is a convenience tool for executing command scripts exported by packages in valos workspace contexts. It is a generalization of 'npx -c' behaviour, adding discoverability, ability to invoke global scripts and the ability to invoke multiple scripts at once using glob matching.
valos-vault-3.1: valma is installed with npm install -g valma or as a package dependency
This installs the global CLI command vlm . At its core valma is a command dispatcher to valma scripts in various command pools.
valos-vault-3.2: valma searches the scripts first from the package.json scripts pool, then from ./node_modules/.bin/depends pool and lastly from the OS-specific variant of /usr/binglobal pool.
As an example typing vlm status in some directory context would forward the command to valma.bin/valma-status first if one exists and falling back to the more generic versions if not. The call eventually resolves at the global /usr/bin/valma-status . Its implementation then calls vlm .status/**/* which calls all scripts which match the glob .status/**/* and are visible on the execution context pools (the scripts called by vlm status are known as valma status scripts).
valos-vault-3.3: A package can export valma scripts using npm package.json bin section and by prefixing the exported name with _vlm_ as usual. These scripts will be available for all packages depending on this package in their depends pool.
Running vlm with no arguments lists all available commands grouped by pool in current directory context.
valos-vault-3.5: valma can be used in programmatic contexts to run valma scripts. When done so, valma must be added as a dependency.
This happens just like with the CLI by using vlm <command> [<args>] . ("npx -c" would be the alternative but it's slow and limited).
valos-vault-3.5.1: valma ensures that node environment is loaded
The environment is loaded only once even for recursive script invokations.
valos-vault-3.5.2: valma ensures that 'vlm' is always found in path
This is so that valma scripts can call 'vlm' even valma is not globally installed as long as valma has been installed as a dependency.
valos-vault-3.5.3: valma ensures that the most specific 'vlm' version is used to evaluate a command, preferring scripts over depended over global.
This is so that toolkits can precisely control the whole toolchain in their dependencies.

§ ValOS `utility` layers provide operational services

ValOS has four main utility layers: files, packages, authorities and chronicles. These layers form the core operational infrastructure of ValOS.

§ Overview of utility layers

valos-vault-4.1.1: An utility is a domain with a well-defined operational purpose.
valos-vault-4.1.2: utility must explicitly define the payload it provides to its consumers as well as the providers, tools and workflows used to manage that payload.
Below is a rough correlation of similar concepts across utilities.

  • Utility- the utility layer which is being described
  • Tool - the name of the tool used to manipulate the payload and/or metadata

  • Payload - the content or the service the utility delivers to consumers

  • Providers - the authoritative source for the payload

  • Consumed via - the mechanism used by a consumer to access the payload

  • Upstream - the possible external source of payload updates

  • Configuration - where the configuration of the utility itself is specified

  • Modifed via - how to make local changes to the payload

  • Produced via - how to request for a set of local changes to be distributed

  • Authority - who accepts and distributes a change request

  • Distributed via - how changes are made live to all consumers

Note that files and chronicles don't have an external upstream and thus these bands are the defining authority of all of their payload.

On the other hand packages and authorities use the files as their external upstream: their payload is generated from the content in git repositories. Making updates to such utility content thus requires:
  1. modifying the corresponding upstream git repository
  2. distributing the git changes (a PR followed with git push stable )

  3. distributing the utility update (publish-packages or deploy-release ).

Step 3 can be automated by tooling in particular domains as a response to particularily formed git repository updates.

§ Files utility layer has files committed in git repositories

git is the industry standard for version managing sets of files in a non-centralized ecosystem. No additional tools are provided because there is no need.
valos-vault-4.2.1: ValOS tools should use git as the files provider.
While github.com is the de facto standard provider and the typical choice it must *not* be _required_.
valos-vault-4.2.2: All git providers must be fully supported by all ValOS tools and libraries.

§ Packages utility layer has shared, versioned, dependable sets of files published as npmjs.com packages

valos-vault-4.3.1: The packages utility payload is npmjs.com packages
These packages can be libraries, toolsets or prebuilt release runtimes - any sets of files really. The raison d'être for packages is when several different consumers depend on the same set of files which are also expected to undergo periodic development. The files thus need versioning, dependency management and automated distribution - this all is provided by npm.

Note: npmjs.com is a javascript repository - this should not be a problem as long as ValOS remains mostly javascript and config files. If a need to diversity the languages arises a private npm registrycan be set up for that purpose.

valma package commands: vlm assemble-packages vlm publish-packages

§ Authorities utility layer has the authority deployments on infrastructure services

valos-vault-4.4.1: A ValOS authority is uniquely identified by an authority URI.
Read more about valos URIs.

valos-vault-4.4.2: A ValOS authority can contain ValOS chronicles and must provide a mechanism for accessing event logs and bvob content as well as for accepting and authorizing incoming commands into authorized chronicle events.
Authorities are usually live deployments on some infrastructure and they provide service APIs as the required mechanisms.

Stateless or in some way non-infrastructural authorities also exist but are specified elsewhere (they are considered degenerate, without upstream and with empty payload).

valos-vault-4.4.3: Authorities utility layer payload (authority payload) is a set of deployed authority service APIs and any associated static content.
The payload here refers to the service deployments and their live APIs themselves and not any dynamic content delivered through them. Such dynamic content belongs to other domains (notably valospace content resides in the chronicles utility layer, see below).

The static content includes HTTP landing pages, site routes and their configurations, ValOS gateway and spindle runtimes and any other similar statically configured files.

valos-vault-4.4.4: An authority may have an operations workspace (opspace) as its upstream for managing its payload.

Particular authorities are naturally free to implement their operational architectures in any way they like. This said opspaces have a well-defined structure which valma authority tools make use of.

Updates to the authority payloads are primarily done as modifications to the corresponding opspace and then distributing those via release deployments.

valos-vault-4.4.5: An opspace should not be published as a package.

While opspaces make use of package.json and the npm dependency management this provides, they can also contain considerable amounts of static content. Also, there should be no reason to depend on an opspace. Automatic release deployment systems should have access to a opspace directly for building the release.

valos-vault-4.4.6: Information must not move from deployed authorities back to authority utility layer upstream.

Information flowing back upstream increases complexity, prevents decentralized and manual upstreams (there is a definite upstream which must be always accessible), and are a security concern (for programmatic access the downstream must have the upstream credentials).

If a use case necessitating this arises, still seriously consider keeping the mutateable content separate from the upstream itself and instead have upstream only contain the necessary code and credentials to access this content.

Note: this applies to architectural decisions and automations only. Interactive content in valospace is not limited from using an opspace to update authorities (although it is still recommended to keep such valospace applications deployments separate from the authorities they are used to control).

valma opspace commands: vlm build-release vlm deploy-release

ValOS core vs. auxiliary authorities

ValOS authorities and any chronicle content they provide do not need to be public. A ValOS core authority is an authority which can be accessed using only ValOS core spindles (including no spindles at all).

A ValOS auxiliary authority is an authority which requires a conforming but non-core ValOS gateway spindle in order to be accessed.

Design Note 2019-03: the spindle conformance requirements are unspecified. When they are specified they must be lenient enough to enable sophisticated protocols but constrained/sandboxed enough that such spindles cannot interfere with other reasonably written spindles.

§ Chronicles utility layer - the foundation of valospace

Event logs and bvob content are the chronicles payload and are consumed by ValOS gateways. It is more extensively covered elsewhere and is mentioned here for completeness; precious little infrastructural tooling is provided for them yet.

Eventually various chronicle diagnostics tools will come in handy:
  • Media content import/export tools
  • Complete chronicle to file system hierarchy save/load tools

  • Event log introspection and manipulation tools

  • etc.

ValOS public vs protected chronicles

All chronicles provided by ValOS authorities are ValOS chronicles. Additionally ValOS public chronicles are chronicles which are both
  1. provided by ValOS core authorities, and
  2. are available for an anonymous consumer with nothing but a client capable of running the ValOS gateway runtime which the authority itself provides (with reasonable concessions for the authority to prevent DDOS attacks)

TODO(iridian): Figure out whether this is actually the most meaningful place to assign this semantic border. A specific term for non-authenticated chronicles capable of running only on standard runtime is useful, but how useful actually?

Kernel domain provides the ValOS primary libraries

It does, indeed (this section pending better understanding on how to write domain specifications).