This document describes how to define the BuildOctree Batch Command for Cogs.Core.Runtime program.
{
"extensions": [
"Cogs.Core.Extensions.RationalReducer",
"Cogs.Core.Extensions.RVM"
],
"source": "Source directory",
"destination": "Destination Directory",
"batch": [],
"properties": {
// List of properties for input parsing as described below.
"XXX": "value"
//
"rvm.tasks": {
// Tasks Properties,
"tasks": [
]
}
},
"post": [
{
"type": "BuildOctree",
"properties": {
// Properties for the "BuildOctree" command.
// Defines number of octree levels, Rational Reducer params for each level etc.
}
},
{
// Exit the Cogs.Runtime program.
// No Exit - Can afterwards open Asset output to check result
"type": "Exit"
}
]
}
Properties sent to the actual loading of the data can be specified at several levels:
First copy all properties from from the Batch-file "properties" section.
"properties": { "keepList": "Path.tags.json", etc.
Then set renamed properties parsed from the Batch-file "properties" section, unless the output property has been set in the Batch property list.
Allows migrating to new batch format setting output properties directly, e.g. Batch properties:
{ "rvm.tagSimplify.keepList": "Path.tags.json" ,,}
Last if "rvm.tasks" is defined, (defining list of RVMParser tasks with properties. See RvmPrser documentation for details).
Overwrite any properties defined in RVMParser tasks definition, e.g:
"rvm.tasks": {
"rvm.tagSimplify.keepList": "My/Custom/Path.tag.json",
}
This allows a user creating a Kognitwin import defintion where only most of the import properties are pre-defined to pass a new "rvm.tasks" def, with modified params, like: "rvm.tessellate.origin": [Custom X/Y/Z]
There are two types of files produced.
Cogsbin files contain geometry, transforms (node hierarchy) and materials.
A cogsbin file can contain multiple root nodes, called parts, and cogs can either load everything or just a part. Each non-empty octree cell corresponds to a part in a cogsbin file, but the a cogsbin file can contain the data of multiple cells (to avoid a ton of tiny requests).
Asset files are JSON files that describe the level-of-detail hierarchy of the resulting model. There are no assumptions here that it is actually an octtree that is encoded, it is just a nested hierarchy of lod-levels, entities and references to other asset files encoded as a tree, so that a subtree is a valid lod hierarchy.
The lod levels are structured with a bounding box, a hierarchy (sometimes
just a single cell) per lod level along with an associated error This
error is the number passed in the level's levelThresholds. See docs in
AssetSystem on how this error is just, but in short it is compared a
tolerance scaled by distance. If the tolerance is 1, the error is just at
what distance the level should be visible.
The process is controlled via numerous variables, the most relevant are:
featureAngle: Minimum angle in radians between normals of two abutting
triangles before their shared edge is considered sharp.
remesh: A non-zero positive value enables remeshing of shapes less thant remesh times tolerance. Defaults to 0.0. Maps to rvm.tessellate.remesh.
remeshGridSpacing: Specifies the remesh grid spacing relative to tolerance. Defaults to 0.125. Maps to rvm.tessellate.remeshGridSpacing. Beware of cubic growth.
remeshVolumeBlur: Specify the number of blur iterations to run on the remesh scalar field. Defaults to 0. Maps to rvm.tessellate.remeshVolumeBlur.
remeshTriSoupBlur: Specify the number of blur iterations to run on the rasterized triangle soups. Defaults to 0. Maps to rvm.tessellate.remeshTriSoupBlur.
remeshParameterization: Specify the tolerance for the simplification run on the remeshed surface. Value is relative to tolerance and defaults to 0.1. Maps to rvm.tessellate.remeshSimplification.
simplificationMode: Controls simplification algorithm used by rvmparser
for triangle soup geometries. Do not set if you do not know what you are
doing.
'levelReducer': Specifies which reduction algorithm that is used for level reduction. The vertexcount-suffix means that the reduction is driven by reducing the vertex count to a percentage of the original vertex count. The alternatives values are:
statistics: If true, run the AssetStatsCommand on
the resulting asset hierarchy.
colorizeLevels: If true, add a material to the output models where the
color denote which lod-level they belong to. This is intended as a debug
tool to make it easier to see where transitions happen. The default is
false where no material is added.
newCode enables the new code path which splits asset files into subtrees
and always re-packs geometry. Default is false.
subtreeSize sets the number of items in an .asset file before branching
off into a new asset subtree. Used to limit size of asset files. Default
value is 100.
params defines a set of input parameterizations, that is rvm files load
with a specific tolerance.
levelParams: defines the number of lod levels, number indexes params to
specify what to use as source data for that level.
levelReduction: one value per level, if nonzero, run Rational Reducer on
finished cell with this value as target percentage.
levelCulling: one value per level, objects smaller than this threshold
will be culled at this level.
levelThresholds: one value per level, specifies the error to write for
that level in the asset-file. If a negative value, error is assumed to be
10 * level.
doNotMergeGeometries do not merge geometries when loading the input data,
required for culling of small objects to work properly.
minCellSize: if a cell gets smaller than this, prune the tree.
keepList list of tags to keep when reading the rvm file, line number of
tag defines object id.
featureAngle: specifies what consitutes a hard edge when rvmparser is
running Rational Reducer on input parameterizations.
batchedFileSize: Merge models up to this file size, reduces number of
small files.
mergeLocalBuffers: Try to pack the geometry of 2x2x2 cell regions into a
single file, to reduce number of small files.
useCompression: Enable ZSTD compression in cogsbin.
compressionLevel: Cogsbin ZSTD compression level.
skipAsset: If true it skips to produce what looks like per-layer asset
files.
forceUpdate: Do not skip processing assets where there already exists an
asset file.
Note: other variables are available and used by RVMLoader (see documentation for rvmparser).
1. The intial step is to figure out the input files. The directory hierarchy
below the path provided in `source` is traversed.
A match pattern is built by concatenating the `prePattern`, `pattern` and
`postPattern` properties. This pattern is split into file extension part
and file stem part.
The extension pattern can either be `*`, which matches all extensions, or
a specific string that must match exactly.
If the stem pattern just contains a `*`, everything matches, and if if the
pattern is `*` and something else, the `*` in the pattern is replaced with
`.*` and the pattern is interpreted as a regular expression. Thus, if the
pattern contains something that can be interpreted as a regular expression
command and contains a `*`, it will be interpreted as that. If the stem
pattern contains no `*` it has to match verbatim.
If both the stem and extension matches the pattern, the file is put into
the list of sources.
2. Then the following steps are processed per source in the list of sources:
2.1 If a matching asset file already exists, processing of that source is
skipped unless `forceUpdate` (default false) is true.
2.2 The source is read multiple times, once per entry in the `params`
array, creating a set of parameterizations, that is a model of the
input with tessellated to a specific degree:
- `params` items set the rvm loader's approximation tolerance.
Forwarded to `rvm.tessellate.tolerance`.
- `paramRemesh`. Optional array of per-parameterization values overriding the value set by `remesh`.
- `paramRemeshBlur`. Optional array of per-parameterization values overriding the value set by `remeshBlur`.
- `paramRemeshSimplification`. Optional array of per-parameterization values overriding the value set by `remeshParameterization`.
- `paramFeatureAngle`. Optional array of per-parameterization values
overriding the value set by `featureAngle`.
- `paramCullDiameter`. Optional array of per param cull diameter thresholds.
Cull round geometries with diameter smaller than threshold.
Enabled with positive values.
Value is relative to params value (tolerance).
Maps to `rvm.tessellate.cullDiameter`.
- `paramCullGeometry`. Optional array of per param cull geometry thresholds.
Culls geometries that are smaller than threshold.
Enabled with positive values.
Value is relative to params value (tolerance).
Maps to `rvm.tessellate.cullGeometryThreshold`.
- `paramCullLeaf`. Optional array of per param cull leaf node thresholds.
Culls leaf nodes where combined geometries are smaller than threshold.
Enabled with positive values.
Value is relative to params value (tolerance).
Maps to `rvm.tessellate.cullLeafThreshold`.
- `paramSimplificationFactor`. Optional array with values that can
enable simplification of non-parametric shapes. A non-positive value
disables simplification and a positive value enables simplfication
within simplificationFactor times approximation tolerance (set in
params).
- `paramSimplificationMode`. Optional arrray specifying per-parameterization
which simplification algorithm to use. Do not set unless you know what
you are doing.
- `keepList` is a path to a newline-separated list of objects to keep.
The input file hierarchy is traversed and if an object is in the keepList,
that object as well as all its children are preserved. This subtree
also gets an object id corresponding to the line number in the keep
list where the object name was. If a kept object has a child that is
also a kept object, the child will get the child's object id, that
it, it works as expected. Forwarded to `rvm.tagSimplify.keepList`.
- `simplify` enables Rational Reducer when loading rvm files, except
for the first parameterization, which have never simplify enabled.
Forwarded to `rvm.simplify`.
- `featureAngle` (default 0.7) specifies the feature angle rational
reducer uses to detect sharp edges. Forwarded to
`rvm.simplify.featureAngle`.
- `doNotMergeGeometries` (default false). Never merge any geometries,
that is, create a new mesh for every geometry in the mesh file. It
also sets the name of the Cogs ModelPart to the kind
of geometry it was ("Pyramid, "Box", "Snout", etc.). Creates a lot of
tiny meshes, but makes it easier to determine the origin of
geometries. Forwarded to `rvm.export.doNotMergeGeometries`.
- For all parameterizations except the first all line segments (often
tiny markers) are discarded. Forwarded to `rvm.export.discardLines`.
--
So, the end result is a set of parameterizations, where one
parameterization is a cogs model sourced from rvm/rvmdump/vue files using a
specific set of import parameters.
2.3 Then the input is analyzed and the octree is set up. The max number
of levels is given by the length of `levelParams`. The value in
`levelParams` specifies which parameterization (built in the previous
step) that should be used a source for this level.
Then the parts of the input model is iterated over. If the name of
the part contains the string passed in `skipPattern`, the part gets
discarded. The scene bounding box is found as well as the size of the
largest geometry.
The actual cells are set up from the bounding box, the max number of
levels, and `minCellSize` which limits the refinement of the tree.
2.4 Then, for each level:
- The set of cells are initialized. A single mesh will be produced by
each cell.
- The parts of the input model are traversed:
- If a part's side length is smaller than that levels
`levelCulling`, it is discarded. Not that the parameterization
parameter `doNotMergeGeometries` probably affects this behavior.
- If `minBounds` is specified, the part is discarded if its bounding
box extends below this bound.
- If `maxBounds` is specified, the part is discarded if its bounding
box extends beyond this bound.
- If `discardOutliers` is enabled (defaults to false), the vertex
count is checked against `outlierMinVertexCount`, vertex
count/volume checked against `outlierMinVertexDensity` and volume
checked against `outlierMaxVolume`. If the part violates all three
checks, it is discarded unless `outputOutliers` is set.
- The sets of cells intersecting the part are found, the parts that
are not fully contained by the cell are tagged for clipping.
- Then for each cell, the intersecting model parts are merged into a
single mesh, parts tagged for clipping are clipped.
2.5 And then Rational Reducer is optionally run the geometry of each cell.
The value from `reductionError` is the target reduction percent.
If level is greater than `dropReductionLevelThreshold` (default 4),
the reduction percent is tapered off if cell vertex count is smaller
than `dropReductionThreshold` (default 1000), down to zero reduction
at vertex count equal to `minReductionThreshold` (default 500).
If the resulting reduction percent is nonzero, Rational Reducer is
applied to the cell's mesh, with `edgeLengthWeight` (default 1)
passed as the edge length parameter. This tells how heavy to weight
edge length relative to model bounding box diagonal (i.e. how hard
should it try to simplify long edges).
2.7 Then the meshes of cells are written to disk, `useCompression`
(default false) enables cogsbin ZSTD compression using
`compressionLevel`(default 0).
There are two strategies to pack the geometry of multiple cells into
a single file:
- `batchedFileSize` (defaults to 1MB): Combine cell models until the
combined file size exceeds `batchedFileSize`.
- `mergeLocalBuffers` (default false): Pack the geometry of 2x2x2
cell regions into a single file and flush after each such region.
Buffers will still be flushed if buffer grows past
`batchedFileSize`.
Levels are processed in order, and then groups of 2x2x2 cells are
processed in z, x, y order.
If packing beyond 2x2x2 cells will be relevant, it might be an idea
to try out traversing the cells in Morton order.
2.8 Finally, the asset file for this source is generated, a JSON files
that encodes the hierarchy.
The levels are traversed, processing the cells building up an
hierarchy where each subtree is represented with a bounding box and N
error thresholds and N subtrees. The error for a subtree is the
level's value in `levelThresholds`, where a positive value is passed
directly, and a negative value adds an automatic error of level * 10.
3. After processing the sources:
if not `skipAsset` (default false) is
true, a single master asset file is created that references the asset
file of each source.
maxSubtreeSize: Max number of items (lod-nodes/models) in a subtree of the
lod-hierarchy before packing it in its own .asset file. Set to 0 to disable,
creating a single large .asset file.maxMeshVertexCount: The number of vertices in a mesh before we split it. Used to
split cells with very much geometry into multiple meshes to avoid stalling when
loading a single huge mesh. Set to 0 to disable, and no meshes will be split
within a cell.maxModelVertexCount: Meshes with more vertices than this will be put into its
own model file. Smaller meshes are packed together in shared model files. Set to 0
to disable, then all meshes will be put in its own cogsbin file (i.e. no shared files).maxModelMeshCount: The max number of meshes to be packed into a shared model file.
Increase this number to reduce the number of small model files, at the expense of loading
unnecessary data. Set to 0 to disable, so there is no limit to the number of meshes
in a cogsbin file.
unnecessary data.embedMeshBounds (default true): Include bounds for select meshes allowing tighter
culling at the expense of more data in the .asset file.