On the fly assemblies in maxscript

In the previous part of the tutorial we read binary data from a file and built a mesh with pure maxscript. In this part we’re going to replace some of the operations with .NET and see if that improves the performance. We’re also going to slice the meshes to avoid having one big mesh. This should improve viewport performance a lot.

This article and part one and three have also appeared on the Artur Leao’s website youcandoitvfx.

Check out the other parts in this tutorial

On the fly assembly

Within 3dsMax you can write with some kind of hybrid maxscript .NET code. While this expands the amount of possibilities of what you can do with scripting, it doesn’t necessarily improve speed. Especially when transferring values between .NET and maxscript stuff can actually slow down. As an alternative you can use an on-the-fly assembly. This works similar to a regular C# assembly you’d call from an external file. The on-the-fly assembly is created in memory only. The advantage is you can run almost pure C# code with all its speed benefits over maxscript. We’ll use that to get the data from the file. This particular piece of code is courtesy of denisT at cgtalk.

Code sample maxscript + .NET

One method has been exchanged since the last part. Data isn’t read anymore from a stream but by an on-the-fly assembly. Also a simple struct has been added and a method to create chunks from a larger grid.

(
    function fn_onTheFlyAssembly_readFileOps =
    (
        /*<FUNCTION>
        Description
            creates an on the fly assembly
            the method reads bytes from a binary file and returns integers. Also takes care of
            converting the data from big endian to little endian
            major parts of this code by denisT: http://forums.cgsociety.org/showpost.php?p=7838323&postcount=6
        Arguments
        Return
        <FUNCTION>*/
        
        --the C# code
        source  = ""
        source += "using System;\n"
        source += "public class ReadFileOps\n"
        source += "{\n"
        source += "    public Int16[] ReadFileShort(string file)\n"
        source += "    {\n"
        source += "        byte[] data = System.IO.File.ReadAllBytes(file);\n"
        source += "        int len = Buffer.ByteLength(data);\n"
        source += "        Int16[] result = new Int16[len / 2];\n"
        source += "        for (int k = 0, i = 0; k < len; k += 2, i++)\n"
        source += "        {\n"
        source += "            result[i] = (Int16)(data[k] << 8 | data[k+1]);\n"
        source += "        }\n"
        source += "        return result;\n"
        source += "    }\n"
        source += "}\n"

        --setting up the assembly in memory
        csharpProvider = dotnetobject "Microsoft.CSharp.CSharpCodeProvider"
        compilerParams = dotnetobject "System.CodeDom.Compiler.CompilerParameters"
        compilerParams.GenerateInMemory = on
        compilerResults = csharpProvider.CompileAssemblyFromSource compilerParams #(source)
        compilerResults.CompiledAssembly.CreateInstance "ReadFileOps"
    )
    
    struct str_chunk
    (
        pos = [0,0], --the position in samples
        segments = [100,100], --the amount of width and length segments for this chunk
        segmentSize = 90, --the size of a single segment. For srtm3 this is 3 arc seconds which is about 90 meters
        theDataIndices = #{} --indices for a data-array. These are the indices corresponding to this chunk. Storing the data itself will just take memory and is redundant
    )

    function fn_initDataGrid &outputmessage slices:5 gridSamples:1201  =
    (
        /*<FUNCTION>
        Description
            builds a grid of data structs based on a datagrid of a particular size
            each datastruct has a size and a position. they're created in such a way that they cover the input gridSamples
            the chunks tile across the datagrid north > south, west > east
        Arguments
            <value by reference> outputmessage: a message we're reporting to
            <integer> slices: the amount of width and length slices we want to slice the input grid into
            <integer> gridSamples: the amount of width and length samples the input grid has
        Return
            <array> an array of structs
        <FUNCTION>*/
            
        --calculate the sizes of the chunks based on the amount of slices you want to split the input grid into
        --adjacent chunks will share vertices
        local chunkSegments = [gridSamples as integer/slices as integer,gridSamples as integer/slices as integer]
        --if we're splitting into slices, add an extra row and column to allow for the overlap
        if slices > 1 do chunkSegments += [1,1]
        --make sure all samples are being used. add them to the chunks at the end of the row and column
        local lastChunkSegments = [chunkSegments.x + (mod (gridSamples-1) slices),chunkSegments.y + (mod (gridSamples-1) slices)]
        format "chunkSegments: %\nLastChunkSize: %\nAmount of chunks: %\n" chunkSegments lastChunkSegments (slices^2) to:outputmessage

        --create and collect the datastructs
        local arrChunkData = #()
        for x = 1 to slices do for y = 1 to slices do
        (
            --build a chunk struct and determine its size and position in the datagrid
            local theData = str_chunk()
            theData.segments.x = if x == slices then lastChunkSegments.x else chunkSegments.x
            theData.segments.y = if y == slices then lastChunkSegments.y else chunkSegments.y
            theData.pos.x = (x-1)*(chunkSegments.x-1)
            theData.pos.y = (y-1)*(chunkSegments.y-1)
            format "\tChunk %, position [%,%]\n" ((x-1)*slices + y) theData.pos.x theData.pos.y to:outputmessage
            
            --create a bitarray which marks the needed data for this chunk from the acquired datagrid
            theData.theDataIndices[gridSamples^2] = false --initialize the bitarray
            for y = 1 to theData.segments.y do
            (
                local startByte = (theData.pos.y+y-1)*gridSamples + theData.pos.x
                for x = 1 to theData.segments.x do
                (
                    theData.theDataIndices[startByte+x] = true
                )
            )
            append arrChunkData theData
        )
        arrChunkData
    )

    function fn_buildMesh theChunk =
    (
        /*<FUNCTION>
        Description
            builds a mesh object from rows and columns of heights. Intended to use with hgt files. this is data also known as srtm.
            uses a datastruct to determine what's being built
        Arguments
            <chunkData struct> theChunk: a datastruct, containing the info needed to create and translate the mesh
            <array> arrHeight: an array of heights as integer
        Return
            <mesh> the created mesh
        <FUNCTION>*/
        
        --build a planar mesh
        local theMesh = Editable_mesh wirecolor:(random (color 30 20 0) (color 30 30 10))
        setMesh theMesh\
            width:((theChunk.segments.x-1)*theChunk.segmentSize)\
            length:-((theChunk.segments.y-1)*theChunk.segmentSize)\ --a negative length puts the first vertex at the top left. This matches nicely with the data
            widthsegs:(theChunk.segments.x-1)\
            lengthsegs:(theChunk.segments.y-1)
        
        --flip the normals because we set the length to a negative value
        addModifier theMesh (Normalmodifier flip:true)
        convertToMesh theMesh
        
        --place the mesh in the right position of the grid
        theMesh.position = [theChunk.pos.x*theChunk.segmentSize,-theChunk.pos.y*theChunk.segmentSize,0]
        
        update theMesh
        forceCompleteRedraw()
        theMesh
    )

    function fn_applyHeights theMesh arrHeight theIndices =
    (
        /*<FUNCTION>
        Description
            applies the heights to the vertices in the mesh
        Arguments
            <mesh object> theMesh: the mesh we're editing
            <array> arrHeight: an array of integers we'll use as heights
            <bitarray> theIndices: a bitarray which marks whichs heights need to be used
        Return
        <FUNCTION>*/
        
        local pos = theMesh.pos
        theMesh.pos = [0,0,0]
        --assigning heights to each vert. We're using the index stored in the indexarray of the chunk to access the correct height value
        local vertIdx = 1
        local meshvert = undefined
        local arrVert = for h in theIndices collect
        (
            meshvert = getVert theMesh vertIdx
            meshvert.z = arrHeight[h]
            vertIdx += 1
            meshvert
        )
        setMesh theMesh vertices:arrvert
        update theMesh
        theMesh.pos = pos
    )
    
    gc()
    local st = timeStamp()
    local mem = heapFree
    local msg = "" as stringstream
    local strFile = @"N:\GitHub\KML for 3dsMax\TestData\S23W068.hgt\S23W068.hgt"
    
    local ReadFileOps = fn_onTheFlyAssembly_readFileOps()
    local arrInt = ReadFileOps.ReadFileShort strFile
    local arrChunkData = fn_initDataGrid &msg slices:3
    for chunk in arrChunkData do
    (
        local theMesh = fn_buildMesh chunk
        fn_applyHeights theMesh arrInt chunk.theDataIndices
    )
    
    format "Time: % ms, memory: %\n" (timestamp()-st) (mem-heapfree)
    format "%" (msg as string)
)

Reading the data

The data is now read with the assembly and is a lot faster. The result is still an array of integers though, so the rest of the pipeline could stay the same. A downside is the data still has to be returned to maxscript variables. This slows down the entire solution.

Chunks

Splitting incoming data into manageable chunks is a common task. I’ve built a small struct to represent a chunk. It’s as if we’re splitting a checkerboard into the separate checkers. Each checker refers to its own piece of the data and has an x/y coordinate. After defining the chunks, we create one mesh for each checker and position it on the board. We also associate the right parts of the data from the file to this chunk. Note that we don’t actually split the array with data into chunks, but just refer to the data with a bitarray. This is much faster and saves memory. After that it’s mostly the same as before. Don’t forget to place the checker in the right position on the board!

Evaluation

This version of the code runs in about 8 seconds which isn’t really better the first iteration. We do have better viewport performance because of the sliced up mesh. Also the data spikes are interpreted correctly now, they lie below the groundplane and don’t mess visually with the data.

Let’s see if we can speed it up by moving more calculations into C#. In the next part we’ll also create a real C# assembly in Visual Studio.

Check out the other parts in this tutorial