@kognifai/cogsengine
    Preparing search index...

    Build an .asset/.cogsbin hierarchy with an octtree structure

    This document describes how to define the BuildOctree Batch Command for Cogs.Core.Runtime program.

    {
    "extensions": [
    "Cogs.Core.Extensions.RationalReducer",
    "Cogs.Core.Extensions.RVM"
    ],
    "source": "Source directory",
    "destination": "Destination Directory",
    "batch": [],
    "properties": {
    // List of properties for input parsing as described below.
    "XXX": "value"
    //
    "rvm.tasks": {
    // Tasks Properties,
    "tasks": [
    ]
    }
    },
    "post": [
    {
    "type": "BuildOctree",
    "properties": {
    // Properties for the "BuildOctree" command.
    // Defines number of octree levels, Rational Reducer params for each level etc.
    }
    },
    {
    // Exit the Cogs.Runtime program.
    // No Exit - Can afterwards open Asset output to check result
    "type": "Exit"
    }
    ]
    }

    Properties sent to the actual loading of the data can be specified at several levels:

    First copy all properties from from the Batch-file "properties" section.

        "properties": { "keepList": "Path.tags.json",  etc.
    

    Then set renamed properties parsed from the Batch-file "properties" section, unless the output property has been set in the Batch property list.

    Allows migrating to new batch format setting output properties directly, e.g. Batch properties:

        { "rvm.tagSimplify.keepList": "Path.tags.json" ,,}
    

    Last if "rvm.tasks" is defined, (defining list of RVMParser tasks with properties. See RvmPrser documentation for details).

    Overwrite any properties defined in RVMParser tasks definition, e.g:

        "rvm.tasks": {
    "rvm.tagSimplify.keepList": "My/Custom/Path.tag.json",
    }

    This allows a user creating a Kognitwin import defintion where only most of the import properties are pre-defined to pass a new "rvm.tasks" def, with modified params, like: "rvm.tessellate.origin": [Custom X/Y/Z]

    There are two types of files produced.

    1. Cogsbin files contain geometry, transforms (node hierarchy) and materials.

      A cogsbin file can contain multiple root nodes, called parts, and cogs can either load everything or just a part. Each non-empty octree cell corresponds to a part in a cogsbin file, but the a cogsbin file can contain the data of multiple cells (to avoid a ton of tiny requests).

    2. Asset files are JSON files that describe the level-of-detail hierarchy of the resulting model. There are no assumptions here that it is actually an octtree that is encoded, it is just a nested hierarchy of lod-levels, entities and references to other asset files encoded as a tree, so that a subtree is a valid lod hierarchy.

      The lod levels are structured with a bounding box, a hierarchy (sometimes just a single cell) per lod level along with an associated error This error is the number passed in the level's levelThresholds. See docs in AssetSystem on how this error is just, but in short it is compared a tolerance scaled by distance. If the tolerance is 1, the error is just at what distance the level should be visible.

    The process is controlled via numerous variables, the most relevant are:

    • featureAngle: Minimum angle in radians between normals of two abutting triangles before their shared edge is considered sharp.

    • remesh: A non-zero positive value enables remeshing of shapes less thant remesh times tolerance. Defaults to 0.0. Maps to rvm.tessellate.remesh.

    • remeshGridSpacing: Specifies the remesh grid spacing relative to tolerance. Defaults to 0.125. Maps to rvm.tessellate.remeshGridSpacing. Beware of cubic growth.

    • remeshVolumeBlur: Specify the number of blur iterations to run on the remesh scalar field. Defaults to 0. Maps to rvm.tessellate.remeshVolumeBlur.

    • remeshTriSoupBlur: Specify the number of blur iterations to run on the rasterized triangle soups. Defaults to 0. Maps to rvm.tessellate.remeshTriSoupBlur.

    • remeshParameterization: Specify the tolerance for the simplification run on the remeshed surface. Value is relative to tolerance and defaults to 0.1. Maps to rvm.tessellate.remeshSimplification.

    • simplificationMode: Controls simplification algorithm used by rvmparser for triangle soup geometries. Do not set if you do not know what you are doing.

    • 'levelReducer': Specifies which reduction algorithm that is used for level reduction. The vertexcount-suffix means that the reduction is driven by reducing the vertex count to a percentage of the original vertex count. The alternatives values are:

      • `rationalreducer-vertexcount': Use Rational Reducer (the default)
      • 'meshoptimizer-vertexcount': Use the simplification code in the meshoptimizer library.
      • 'meshoptimizer-sloppy-vertexcount': Use the sloppy simplification code in the meshoptimized library that doesn't try to preserve topology.
    • statistics: If true, run the AssetStatsCommand on the resulting asset hierarchy.

    • colorizeLevels: If true, add a material to the output models where the color denote which lod-level they belong to. This is intended as a debug tool to make it easier to see where transitions happen. The default is false where no material is added.

    • newCode enables the new code path which splits asset files into subtrees and always re-packs geometry. Default is false.

    • subtreeSize sets the number of items in an .asset file before branching off into a new asset subtree. Used to limit size of asset files. Default value is 100.

    • params defines a set of input parameterizations, that is rvm files load with a specific tolerance.

    • levelParams: defines the number of lod levels, number indexes params to specify what to use as source data for that level.

    • levelReduction: one value per level, if nonzero, run Rational Reducer on finished cell with this value as target percentage.

    • levelCulling: one value per level, objects smaller than this threshold will be culled at this level.

    • levelThresholds: one value per level, specifies the error to write for that level in the asset-file. If a negative value, error is assumed to be 10 * level.

    • doNotMergeGeometries do not merge geometries when loading the input data, required for culling of small objects to work properly.

    • minCellSize: if a cell gets smaller than this, prune the tree.

    • keepList list of tags to keep when reading the rvm file, line number of tag defines object id.

    • featureAngle: specifies what consitutes a hard edge when rvmparser is running Rational Reducer on input parameterizations.

    • batchedFileSize: Merge models up to this file size, reduces number of small files.

    • mergeLocalBuffers: Try to pack the geometry of 2x2x2 cell regions into a single file, to reduce number of small files.

    • useCompression: Enable ZSTD compression in cogsbin.

    • compressionLevel: Cogsbin ZSTD compression level.

    • skipAsset: If true it skips to produce what looks like per-layer asset files.

    • forceUpdate: Do not skip processing assets where there already exists an asset file.

    Note: other variables are available and used by RVMLoader (see documentation for rvmparser).

    1. The intial step is to figure out the input files. The directory hierarchy
        below the path provided in `source` is traversed.
    
        A match pattern is built by concatenating the `prePattern`, `pattern` and
        `postPattern` properties. This pattern is split into file extension part
        and file stem part.
    
        The extension pattern can either be `*`, which matches all extensions, or
        a specific string that must match exactly.
    
        If the stem pattern just contains a `*`, everything matches, and if if the
        pattern is `*` and something else, the `*` in the pattern is replaced with
        `.*` and the pattern is interpreted as a regular expression. Thus, if the
        pattern contains something that can be interpreted as a regular expression
        command and contains a `*`, it will be interpreted as that. If the stem
        pattern contains no `*` it has to match verbatim.
    
        If both the stem and extension matches the pattern, the file is put into
        the list of sources.
    
    2. Then the following steps are processed per source in the list of sources:
    
        2.1 If a matching asset file already exists, processing of that source is
            skipped unless `forceUpdate` (default false) is true.
    
        2.2 The source is read multiple times, once per entry in the `params`
            array, creating a set of parameterizations, that is a model of the
            input with tessellated to a specific degree:
    
            - `params` items set the rvm loader's approximation tolerance.
              Forwarded to `rvm.tessellate.tolerance`.
    
            - `paramRemesh`. Optional array of per-parameterization values overriding the value set by `remesh`.
    
            - `paramRemeshBlur`. Optional array of per-parameterization values overriding the value set by `remeshBlur`.
    
            - `paramRemeshSimplification`. Optional array of per-parameterization values overriding the value set by `remeshParameterization`.
    
            - `paramFeatureAngle`. Optional array of per-parameterization values
               overriding the value set by `featureAngle`.
    
            - `paramCullDiameter`. Optional array of per param cull diameter thresholds.
              Cull round geometries with diameter smaller than threshold.
              Enabled with positive values.
              Value is relative to params value (tolerance).
              Maps to `rvm.tessellate.cullDiameter`.
    
            - `paramCullGeometry`. Optional array of per param cull geometry thresholds.
              Culls geometries that are smaller than threshold.
              Enabled with positive values.
              Value is relative to params value (tolerance).
              Maps to `rvm.tessellate.cullGeometryThreshold`.
    
            - `paramCullLeaf`. Optional array of per param cull leaf node thresholds.
              Culls leaf nodes where combined geometries are smaller than threshold.
              Enabled with positive values.
              Value is relative to params value (tolerance).
              Maps to `rvm.tessellate.cullLeafThreshold`.
    
            - `paramSimplificationFactor`. Optional array with values that can
              enable simplification of non-parametric shapes. A non-positive value
              disables simplification and a positive value enables simplfication
              within simplificationFactor times approximation tolerance (set in
              params).
            
            - `paramSimplificationMode`. Optional arrray specifying per-parameterization
              which simplification algorithm to use. Do not set unless you know what
              you are doing.
    
            - `keepList` is a path to a newline-separated list of objects to keep.
    
              The input file hierarchy  is traversed and if an object is in the keepList,
              that object as well as all its children are preserved. This subtree
              also gets an object id corresponding to the line number in the keep
              list where the object name was. If a kept object has a child that is
              also a kept object, the child will get the child's object id, that
              it, it works as expected. Forwarded to `rvm.tagSimplify.keepList`.
    
            - `simplify` enables Rational Reducer when loading rvm files, except
              for the first parameterization, which have never simplify enabled.
              Forwarded to `rvm.simplify`.
    
            - `featureAngle` (default 0.7) specifies the feature angle rational
              reducer uses to detect sharp edges. Forwarded to
              `rvm.simplify.featureAngle`.
    
            - `doNotMergeGeometries` (default false). Never merge any geometries,
              that is, create a new mesh for every geometry in the mesh file. It
              also sets the name of the Cogs ModelPart to the kind
              of geometry it was ("Pyramid, "Box", "Snout", etc.). Creates a lot of
              tiny meshes, but makes it easier to determine the origin of
              geometries. Forwarded to `rvm.export.doNotMergeGeometries`.
    
            - For all parameterizations except the first all line segments (often 
              tiny markers) are discarded. Forwarded to `rvm.export.discardLines`.
              --
              So, the end result is a set of parameterizations, where one
              parameterization is a cogs model sourced from rvm/rvmdump/vue files using a
              specific set of import parameters.
    
        2.3  Then the input is analyzed and the octree is set up. The max number
             of levels is given by the length of `levelParams`. The value in
             `levelParams` specifies which parameterization (built in the previous
             step) that should be used a source for this level.
    
             Then the parts of the input model is iterated over. If the name of
             the part contains the string passed in `skipPattern`, the part gets
             discarded. The scene bounding box is found as well as the size of the
             largest geometry.
    
             The actual cells are set up from the bounding box, the max number of
             levels, and `minCellSize` which limits the refinement of the tree.
    
        2.4 Then, for each level:
    
            - The set of cells are initialized. A single mesh will be produced by
              each cell.
    
            - The parts of the input model are traversed:
    
              - If a part's side length is smaller than that levels
                `levelCulling`, it is discarded. Not that the parameterization
                parameter `doNotMergeGeometries` probably affects this behavior.
    
              - If `minBounds` is specified, the part is discarded if its bounding
                box extends below this bound.
    
              - If `maxBounds` is specified, the part is discarded if its bounding
                box extends beyond this bound.
    
              - If `discardOutliers` is enabled (defaults to false), the vertex
                count is checked against `outlierMinVertexCount`, vertex
                count/volume checked against `outlierMinVertexDensity` and volume
                checked against `outlierMaxVolume`. If the part violates all three
                checks, it is discarded unless `outputOutliers` is set.
    
              - The sets of cells intersecting the part are found, the parts that
                are not fully contained by the cell are tagged for clipping.
    
              - Then for each cell, the intersecting model parts are merged into a
                single mesh, parts tagged for clipping are clipped.
    
        2.5  And then Rational Reducer is optionally run the geometry of each cell.
             The value from `reductionError` is the target reduction percent.
    
             If level is greater than `dropReductionLevelThreshold` (default 4),
             the reduction percent is tapered off if cell vertex count is smaller
             than `dropReductionThreshold` (default 1000), down to zero reduction
             at vertex count equal to  `minReductionThreshold` (default 500).
    
             If the resulting reduction percent is nonzero, Rational Reducer is
             applied to the cell's mesh, with `edgeLengthWeight` (default 1)
             passed as the edge length parameter. This tells how heavy to weight
             edge length relative to model bounding box diagonal (i.e. how hard
             should it try to simplify long edges).
    
        2.7  Then the meshes of cells are written to disk, `useCompression`
             (default false) enables cogsbin ZSTD compression using
             `compressionLevel`(default 0).
    
              There are two strategies to pack the geometry of multiple cells into
              a single file:
    
              - `batchedFileSize` (defaults to 1MB): Combine cell models until the
                combined file size exceeds `batchedFileSize`. 
    
              - `mergeLocalBuffers` (default false): Pack the geometry of 2x2x2
                cell regions into a single file and flush after each such region.
                Buffers will still be flushed if buffer grows past
                `batchedFileSize`.
    
             Levels are processed in order, and then groups of 2x2x2 cells are
             processed in z, x, y order.
    
             If packing beyond 2x2x2 cells will be relevant, it might be an idea
             to try out traversing the cells in Morton order.
    
        2.8  Finally, the asset file for this source is generated, a JSON files
             that encodes the hierarchy. 
    
             The levels are traversed, processing the cells building up an
             hierarchy where each subtree is represented with a bounding box and N
             error thresholds and N subtrees. The error for a subtree is the
             level's value in `levelThresholds`, where a positive value is passed
             directly, and a negative value adds an automatic error of level * 10.
    
    3. After processing the sources:
    
        if not `skipAsset` (default false) is
          true, a single master asset file is created that references the asset
          file of each source.
    
    • maxSubtreeSize: Max number of items (lod-nodes/models) in a subtree of the lod-hierarchy before packing it in its own .asset file. Set to 0 to disable, creating a single large .asset file.
    • maxMeshVertexCount: The number of vertices in a mesh before we split it. Used to split cells with very much geometry into multiple meshes to avoid stalling when loading a single huge mesh. Set to 0 to disable, and no meshes will be split within a cell.
    • maxModelVertexCount: Meshes with more vertices than this will be put into its own model file. Smaller meshes are packed together in shared model files. Set to 0 to disable, then all meshes will be put in its own cogsbin file (i.e. no shared files).
    • maxModelMeshCount: The max number of meshes to be packed into a shared model file. Increase this number to reduce the number of small model files, at the expense of loading unnecessary data. Set to 0 to disable, so there is no limit to the number of meshes in a cogsbin file. unnecessary data.
    • embedMeshBounds (default true): Include bounds for select meshes allowing tighter culling at the expense of more data in the .asset file.