Experimental Specifications
mmmvxspec_1: Project API Specification
(is DEPRECATED)
This specification is DEPRECATED.
Situation Description, Observations and some Ideas
-
Software projects have dependencies that each form a tree of dependencies.
-
A software developer (hereafter: dev_A)
is developing multiple projects
(hereafter: setofprojects_S1) that
have a common dependency (hereafter: dep_X1),
where the software developer is among one of the main developers.
-
If the setofprojects_S1 elements are run/tested on the machine of
dev_A, they should all use a single, sandbox,
version of the dep_X1 by having the sandbox version
of the dep_X1 included to the setofprojects_S1 element
(hereafter: an_element_E1)
folder via a symlink, while the rest of the
an_element_E1 dependencies are in the form
of non-symlinks, folders.
-
dev_A publishes setofprojects_S1 elements by
replacing the dep_X1 symlinks with copies of
the local, sandbox, version of the dep_X1.
To save storage space and to save upload/download time,
the copy can contain relative symlinks to
files/folders within the an_element_E1 project folder.
For example, if an_element_E1 dependencies
dep_X42 and dep_X22 both have the dep_X1 as their dependency,
then the dep_X42 and dep_X22 should use a relative symlink
to the dep_X1 in stead of containing their own full copies
of the dep_X1. The use of relative symlinks should take
place recursively within the whole tree of dependencies.
An illustration:
The algorihtm for transforming a symlink-less an_element_E1
dependency tree to the state that is described in the
Diagram 1
is that the build system of the an_element_E1 receives a hash-table.
The data format of the hash-table is that its
keys are package names
and the
values are full paths to a symlink or a folder that
either contains or references the packages
.
In the case of the Diagram 1 the hash-table
that the build system of the an_element_E1 receives is empty, but it might contain one key-value
pair, where the key is the dep_X1 and the value is the full path to the
sandbox version of the dep_X1 at the machine of the dev_A. The build system of
the an_element_E1 compares the list of an_element_E1 immediate dependencies,
which is dep_X1, dep_42, dep_22, with the list in the hash-table and
replaces non-recursively those immediate dependencies
that are listed in the hash-table, with symlinks to the paths that
are listed in the hash-table. The replacement is carried out even if
the immediate dependency is in a form of a symlink before the replacement.
Then the build system of the an_element_E1 creates a new, empty,
hash-table, lists its own immediate dependencies with the full
paths that reside in the project tree of the an_element_E1 and
adds to that hash-table only those key-value pairs from the
received hash-table that do not depict the immediate dependencies
of the an_element_E1. The build system passes the newly created
hash-table to the build systems of all of those immediate dependencies
of the an_element_E1 that were not listed in the hash-table that
the build system of the an_element_E1 received and
the whole process is repeated recursively.
In the case of this example the algorithm leaves one
redundant copy
of the dep_UFO_1 to the project tree of the an_element_E1. That
inefficiency is eliminated by creating a package,
optimization_hack_1,
that has the an_element_E1 as its immediate dependency.
Then the optimization_hack_1 dependencies that have more than one folder,
regardless of whether they are remote or immediate dependencies of
the optimization_hack_1, are copied to be immediate dependencies of
the optimization_hack_1. After the copying the previously described
symlink creation algorithm is run. The
optimized result
for the case at the Diagram 1:
-
To allow different projects to use different versions of
a single library, the Project API should be
shell/bash based, because that separates
the different versions of the library to
distinct operating system processes.
-
There should be at least some mechanism to
get back errors that happen at some
deeper recursion level. The errors should
be human-readable and machine-readable, not
necessarily at the same time.
The API
The general idea is to use the
Decorator Design Pattern
so that the Project API calls recurse down the dependency tree.
To avoid dependency version collisions in Project API implementations,
the Project API calls are run in separate operating system
processes. The API is console/bash/shell based.
some_application_possibly_rake_or_bashscript command_name command_parameters
where in some cases the implementation is simplified by combining
command_name and command_parameters
to separate, new, commands. For example, if there were a
parametrized command "move", then in stead of
move left
move right
move backwards
the command "move" would be replaced with 3 separate commands that do not
take any parameters:
move_left
move_right
move_backwards
In the case of the Project API one of the command parameters determines, whether the result is printed
to console in human readable form or in some structured text format, like JSON, YAML,
ProgFTE, etc.
Proposed Project API in
EBNF:
API ::= API_RUNNER COMMAND
API_RUNNER ::= Rakefile | bashscript
COMMAND ::= COMMAND_FOR_HUMANREADABLE_OUTPUT | COMMAND_FOR_STRUCTURED_TEXT_OUTPUT
COMMAND_FOR_HUMANREADABLE_OUTPUT ::= COMMANDS_H_1 | COMMANDS_H_2
COMMANDS_H_1 ::= mmmvxspec_1_optimize_storage_recursively_t1 FILE_PATH_2? |
mmmvxspec_1_unoptimize_storage_recursively_t1 FILE_PATH_2?
COMMANDS_H_2 ::= mmmvxspec_1_recreate_project_specific_symlinks_recursively_t1
COMMAND_FOR_STRUCTURED_TEXT_OUTPUT ::= COMMANDS_TX_1 | COMMANDS_TX_2 | COMMANDS_TX_3 | COMMANDS_TX_4 |
COMMANDS_TX_5
COMMANDS_TX_1 ::= mmmvxspec_1_optimize_storage_recursively_t1_progfte FILE_PATH_2? |
mmmvxspec_1_unoptimize_storage_recursively_t1_progfte FILE_PATH_2?
COMMANDS_TX_2 ::= mmmvxspec_1_recreate_project_specific_symlinks_recursively_t1_progfte
COMMANDS_TX_3 ::= mmmvxspec_1_folder2symlink_t1_without_recursion_progfte FILE_PATH_1 |
mmmvxspec_1_symlink2folder_t1_without_recursion_progfte FILE_PATH_1
COMMANDS_TX_4 ::= mmmvxspec_1_get_list_of_immediate_dependencies_t1_progfte |
mmmvxspec_1_get_list_of_folders_of_dependencies_recursively_t1_progfte
COMMANDS_TX_5 ::= mmmvxspec_1_convert_relative_symlinks_2_absolute_symlinks_recursively_t1_progfte |
mmmvxspec_1_convert_absolute_symlinks_2_relative_symlinks_recursively_t1_progfte
The
FILE_PATH_1
is a full path to an "upstream" version
The
FILE_PATH_2
is a full path to a temporary text-file that contains
ProgFTE
of the following hash-table:
ht
x+--[<name of the folder or symlink that holds the dependency in project lib folder >]
=>
< full path to the "upstream" location of the folder or symlink>
The
mmmvxspec_1_get_list_of_immediate_dependencies_t1_progfte
includes both, symlinks and folders, to the list prints
the list to console as ProgFTE text that uses the
same hash-table data format as is used in the file that is referenced by the FILE_PATH_2.
The
mmmvxspec_1_get_list_of_folders_of_dependencies_recursively_t1_progfte
does not include symlinks to the list and returns its results by printing ProgFTE to console.
The result of the mmmvxspec_1_get_list_of_folders_of_dependencies_recursively_t1_progfte
has the following hash-table data format:
ht
x+--[<name of the folder or symlink that holds the dependency in project lib folder >]
=>
< ProgFTE of a hash-table that lists full paths by placing them at keys "0", "1", "2", ...>
API_RUNNER Path
The API_RUNNER is executed by changing the working directory to the path of the API_RUNNER.
The API_RUNNER is searched according to the following precedence:
-
$PROJECT_ROOT/src/dev_tools/Rakefile
-
$PROJECT_ROOT/bonnet/project_api/mmmvxspec_1/apirunner.bash
-
$PROJECT_ROOT/src/bonnet/project_api/mmmvxspec_1/apirunner.bash
-
Throw an exception and stop. All packages that are listed as an immediate dependency of any other
package are expected to have an API_RUNNER.
Additional Notes and Comments
According to the current scheme different versions of the a same software project are expected to be
represented as different packages. That allows substantial API changes within the package and it allows
the different versions to be simultaneously present in the tree of dependencies. Characters that
have special meaning in
Bash
are not allowed to be part of package names. That includes spaces, semicolons, parenthesis, etc.
If the API_RUNNER implementation creates the symlinks with absolute paths, then the symlinks will
probabilistically become dead whenever the optimization_hack_1 is copied from one computer to another. One way
to "fix" that might be to re-run the symlink creation routine on optimization_hack_1 before starting
to use the optimization_hack_1, but that will not work if some of the API_RUNNER dependencies are referenced
by the very same, dead, symlinks that the API_RUNNER is expected to overwrite. Therefore
the symlinks that reference packages within the optimization_hack_1 must be relative symlinks,
because that way the API_RUNNER should, hopefully, stay functional. As relative symlinks probabilistically
break, if their location is changed relative to their target,
the relative symlinks must be converted to
absolute symlinks prior to the switch between folders and symlinks.
In step by step fashion:
mmmvxspec_1_optimize_storage_recursively_t1
-
xxxxconvert relative symlinks to absolute symlinks
-
xxxxapply the recursive folders 2 symlinks conversion
algorithm
-
xxxxconvert absolute symlinks to relative symlinks
mmmvxspec_1_unoptimize_storage_recursively_t1
-
xxxxconvert relative symlinks to absolute symlinks
-
xxxxcopy folders to replace symlinks till no copying takes
place
-
xxxxdo nothing, because there are no absolute symlinks left
to convert to relative symlinks
Some Related References