libgpac
Documentation of the core library of GPAC
Shader Interface Reference

import"evg.idl";

Public Member Functions

void push ()
 
long push (DOMString left_val, DOMString operand, DOMString right_val, optional DOMString right_val2=null)
 
long push (DOMString cond_val, DOMString left_val, DOMString operand, DOMString right_val)
 
long push (DOMString end_cond_val)
 
long push (DOMString goto_val, long stack_index)
 
long push (DOMString goto_val, DOMString stack_index_uniform)
 
void update ()
 

Detailed Description

Shader object common to vertex and fragment shaders

Although QuickJS is an amazing piece of software, calling JavaScript for each fragment is just too costly. The shader object provides a simple set of tools to produce a vertex or fragment output at reasonnable speed, without interacting with JavaScript.

Generic principles

The working principles of the shader object is as follows:

  • no complex language processing (eg GLSL) and basic validation only (at least for the time being)
  • instructions and variables are pushed onto a stack of instructions
  • native EVG objects (Matrix, VertexAttrib, VertexAttribInterpolator, Texture) are stored as pointers to the native objects
  • variables can be defined as uniforms, and updated whenever needed
  • upon executing the shader, the instructions are processed in order

Local variables are untyped, except for reserved variables. The type is infered whenever the variable is assigned or modified, thus a same variable can change type during the execution of the shader. Upon assigning, the type of the source variable is inferred as follows:

  • Javascript bool and integer are converted to integer type
  • Javascript float is converted to float type
  • A javascript object with properties {x, y, z, q/w} or {r,g,b,a} is converted to a 4 component vector
  • A javascript object with properties {x, y, z} or {r,g,b} is converted to a 3 component vector
  • A javascript object with properties {x, y} or {r,g} or {s,t} is converted to a 2 component vector
  • A javascript array is converted to a 1, 2, 3 or 4 float vector
  • A Matrix object is kept as a pointer
  • A VertexAttribInterpolator object is converted to a 2D, 3D or 4D vector depending on the number of components in the object
  • A Texture sample is converted to a 4D vector
Note
There is no support for structures nor arrays in the shader instructions.
Warning
A local variable can only be of type bool, integer, float or vector of float (up to 4 dimension). It is not possible to store an EVG object (Matrix, VertexAttrib, VertexAttribInterpolator, Texture) as a local variable.

The assignment of vectors can use masking of components, using a combination of x, y, z, q (or r, g, b, a or s,`t) values:

shader.push('foo', '=', 'val.xy');
shader.push('foo.xy', '=', 'val.xz');
Warning
there is no swizzling , .xy is equivalent to .yx

Fragment shader reserved variables

These variables names are reserved in a fragment shader, and their type cannot be modified:

Variable name type Semantic
fragColor Vec4f the fragment output color in RGBA colorspace
fragRGBA Vec4f the fragment output color in RGBA colorspace
fragYUVA Vec4f the fragment output color in YUVA colorspace
fragX float the fragment X coordinate
fragY float the fragment Y coordinate
fragDepth float the fragment depth value - can be overwritten by shader
fragZ float the fragment Z coordinate
fragW float the fragment W coordinate
txCoord float the texture coordinates for 2D shaders
txCoordi ints the integer texture coordinates for 2D shaders
fragOdd boolean the odd/even flag of 2D path for 2D shaders
Note
txCoordi is used to fetch the pixel value as int. This is quite experimental, and should only be used to assign fragRGBA or fragYUVA.

Vertex shader reserved variables

These variables names are reserved in a vertex shader, and their type cannot be modified:

Variable name type Semantic
vertex Vec4f the input vertex coordinates
vertexOut Vec4f the output vertex coordinates

Uniforms variables

Uniforms variables are variables whose values need to be updated but do not require a change in the shader logic. Since some of these variables may be native JS interpreter types, they cannot be kept by reference in the stack. Instead, these variables are passed by names, and correspond to properties of the shader object. These properties are evaluated whenever the Shader::update function is called.

Uniforms are referenced in the shader as string variables prefixed with '.'; for example '.foo' will correspond to the 'foo' property of the shader object. The associated JS values can be:

  • boolean
  • integer
  • float
  • array of float, which is converted to Vec2f, Vec3f or Vec4f
  • Float32Array or ArrayBuffer, which is converted to Vec2f, Vec3f or Vec4f

Example:

//shader creation part
...
shader.push('my_var', '=', '.somevar');
... do something with my_var
shader.push('fragRGBA', '=', 'myvar');
//setup draw
canvas.fragment = shader
shader.somevar = [0.5, 0.5, 0.5, 1.0];
shader.update();
//draw

Operations

Operations take zero, one or two right values and assign the result to a left value.

The left value can only be:

  • a built-in variable or a local variable in a fragment shader.
  • a built-in variable, a local variable or a VertexAttribInterpolator in a vertex shader.

The first right value can be a built-in variable, a local variable, a uniform, a Texture, a VertexAttrib (vertex shader only), a VertexAttribInterpolator (fragment shader only) or a Matrix.

Note
There is currently no support for matrix product in the shaders

Single-dimension operations on vectors are done component-wise, eg res.x = sin(rv1.x), res.y = sin(rv1.y), ...

Note
in the following table, rv1 means first right value, rv2 means second right value if present for the operation

Whenever rv2 is allowed, this can only be an already locally defined variable, and cannot be a uniform or a JS Object.

Whenever rv2 is allowed, this can only be an already locally defined variable, and cannot be a uniform or a JS Object.

Both the left value and the right value can be assigned a component mask:

Name rv1 rv2 Semantic
'=' any none assigns rv1 to left value (1)
'*=' any none multiplies left value by rv1 (1)
'*=' Matrix none multiplies left value by the given matrix
'/=' any none divides left value by rv1 (1)
'+=' any none adds right value to rv1 (1)
'=!' any none assigns not rv1 to left value (left = !right) (1)
'normalize' Vec3f none assigns normalize value of right vector to left value
'length' Vec3f none assigns length of right vector to left value (float type)
'distance' Vec3f Vec3f assigns distance between the two right values to left value (float type)
'dot' Vec3f Vec3f assigns dot product of rv1 by rv2 to left value (float type)
'cross' Vec3f Vec3f assigns cross product of rv1 by rv2 to left value (Vec3f)
'pow' any same as rv1 assigns pow of rv1 by rv2 to left value
'sin' any none assigns sin(rv1) to left value
'asin' any none assigns asin(rv1) to left value
'cos' any none assigns cos(rv1) to left value
'acos' any none assigns acos(rv1) to left value
'tan' any none assigns tan(rv1) to left value
'atan' any any assigns atan(rv1, rv2) to left value
'log' any none assigns log(rv1) to left value
'exp' any none assigns exp(rv1) to left value
'log2' any none assigns log2(rv1) to left value
'exp2' any none assigns exp2(rv1) to left value
'sinh' any none assigns sinh(rv1) to left value
'cosh' any none assigns cosh(rv1) to left value
'sqrt' any none assigns sqrt(rv1) to left value
'inversesqrt' any none assigns 1/sqrt(rv1) to left value
'abs' any none assigns absolute value (ABS) of rv1 to left value
'sign' any none assigns sign vector of rv1 to left value - cf GLSL
'floor' any none assigns floor(rv1) to left value
'ceil' any none assigns ceil(rv1) to left value
'fract' any none assigns fractional part of rv1 to left value - cf GLSL
'mod' any any assigns modulo of rv1 by rv2 to left value - cf GLSL
'min' any any assigns minimum of rv1 and rv2 to left value - cf GLSL
'max' any any assigns maximum of rv1 and rv2 to left value - cf GLSL
'clamp' any any clamps left value with rv1 as minimum and rv2 as maximum
'sampler' Texture Vec2f sets left value to the pixel value of the texture rv1 at coordinate rv2, as an RGBA Vec4f
'samplerYUV' Texture Vec2f sets left value to the pixel value of the texture rv1 at coordinate rv2, as a YUVA Vec4f
'discard' none none discards the current fragment and exits shader processing (fragment shader only)
'print' any none prints the given variable rv1 to stderr
'toRGB' Vec3f, Vec4f none converts rv1 to RGB
'toYUV' Vec3f, Vec4f none converts rv1 to YUV

(1): these operations accept component masks on both left value and right value. Example

shader.push('myval.xy', '*=' 'otherval.zq');

If the right value has less component indicated in the mask than left value components, the last indicated component of the right value will be used for the remaining left value components. Example

//myval will be assigned to {x:otherval.z, y:otherval.q, z:otherval.q}
shader.push('if', 'myval.xyz', '=' 'otherval.zq');

Conditions

Conditions are expressed with the following keywords:

  • 'if' : takes 3 additionnal parameters left, operandand right and evaluates the condition
  • 'elseif' : takes 3 additionnal parameters left, operand and right and evaluates the condition
  • 'else': no additional parameters
  • 'end': no additional parameters

Conditions can be nested.

Name Right Value Semantic
'<' any evaluate to true if left is strictly less than right
'<=' any evaluate to true if left is less than or equal to right
'>' any evaluate to true if left is strictly greater than right
'>=' any evaluate to true if left is greater than or equal to right
'==' any evaluate to true if left is equal to right
'!=' any evaluate to true if left is different from right

Example:

...
shader.push('if', 'my_var.x', '<=', 0.5);
shader.push('if', 'my_var.y', '>=', 0.5);
...
shader.push('end');
shader.push('elseif', 'my_var.x', '<=', '.uniform');
...
shader.push('else');
...
shader.push('end');
...

Conditions accept component masks on both left value and right value, resulting in a per-component of the mask test. Example

shader.push('if', 'myval.xy', '<' 'otherval.zq');

If the right value has less component indicated in the mask than left value components, the last indicated component of the right value will be used for the remaining left value components. Example

//myval will be tested against {x:otherval.z, y:otherval.q, z:otherval.q}
shader.push('if', 'myval.xyz', '<' 'otherval.zq');

Stack jumping

The goto statement allows jumping in the stack. The goto statement takes a single parameter which can either be an integer or a uniform name resolving to an integer. This integer gives the 1-based index of the instruction to jump to. Example:

...
shader.label_start = shader.push('if', 'my_var.x', '<=', 0.5);
shader.push('if', 'my_var.y', '>=', 0.5);
...
shader.push('goto', '.label_end'); //go to the instruction at position label_end
...
shader.push('end');
shader.push('elseif', 'my_var.x', '<=', '.uniform');
shader.push('goto', '.label_start'); //go to the instruction at position label_start
shader.push('else');
shader.push('goto', 0); //go to beginning of the instruction stack
shader.label_end = shader.push('end');
...

The value of the jump position is not checked while assigning the goto instruction, it is checked while processing the fragment: this allows jumping anywhere in the stack.

Warning
You must make sure that your shader state is correct when jumping in the stack of instructions !
Note
There is no provision for while and for statements in the shader. This can however be emulated with the goto statement.

Vertex attribute interpolation

The rasterizer performs perspective-correct interpolation of attributes, and provides two objects, VertexAttrib and VertexAttribInterpolator, to handle interpolation. The attribute interpolation can be done:

  • at the fragment shader stage only
  • at both vertex and fragment shader stages

Fragment-shader only interpolation

In this configuration, the attributes to interpolate do not depend on the state of the vertex shader (eg for example vertices colors or texture coordinates). To interpolate the attribute, a VertexAttribInterpolator object is used to wrap the attributes to interpolate.

Example:

let txi = new evg.VertexAttribInterpolator(cube_txcoords, 2, GF_EVG_VAI_VERTEX);
frag_shader=canvas.new_shader(GF_EVG_SHADER_FRAGMENT);
frag_shader.push('txc', '=', txi);
frag_shader.push('txc', '*=', 2);
frag_shader.push('txval', 'sampler', texture, 'txc');
frag_shader.push('fragColor', '=', 'txval');

Per-vertex interpolation

In this configuration, the attributes to interpolate depend on the state of the vertex shader. To interpolate the attribute, a VertexAttrib object is used to wrap the attributes to interpolate at the vertex shader, and a VertexAttribInterpolator object is used to perform the interpolation at the fragment shader.

Example:

let ni = new evg.VertexAttribInterpolator(3);
ni.normalize=true;
let normals = new evg.VertexAttrib(cube_normals_face, 3, GF_EVG_VAI_PRIMITIVE);
vert_shader=canvas.new_shader(GF_EVG_SHADER_VERTEX);
vert_shader.push('normal', '=', normals);
vert_shader.push('normal', '*=', normal_matrix);
vert_shader.push(ni, '=', 'normal');
vert_shader.push('vertex', '*=', matrix);
vert_shader.push('vertexOut', '=', 'vertex');
frag_shader=canvas.new_shader(GF_EVG_SHADER_FRAGMENT);
frag_shader.push('normal', '=', ni);
frag_shader.push('normal.q', '=', 1.0);
frag_shader.push('fragColor', '=', 'normal');

RGB and YUV color spaces

The rasterizer can work on both RGB and YUV output color buffers.

  • When a fragment output value is given in RGB and the output color buffer is in YUV, conversion is performed
  • When a fragment output value is given in YUV and the output color buffer is in RGB, conversion is performed
  • Otherwise, no conversion is done

Texture sampling can also work on both RGB and YUV texture maps.

  • When a sampler call is made on a YUV texture, conversion is performed
  • When a samplerYUV call is made on a RGB texture, conversion is performed
  • Otherwise, no conversion is done

The toRGB and toYUV operations can also be used to convert between color space, but precomputed values should be prefered.

Member Function Documentation

◆ push() [1/6]

void Shader::push ( )

resets the stack of instructions

◆ push() [2/6]

long Shader::push ( DOMString  left_val,
DOMString  operand,
DOMString  right_val,
optional DOMString  right_val2 = null 
)

pushes a new instruction on the stack

Parameters
left_valthe name of the left value to be assigned or modified
operandthe name of the operand to use
right_valthe name of the right value to use by the operand
right_val2the name of the second right value to use by the operand, if any
Returns
the 1-based index of the added instruction

◆ push() [3/6]

long Shader::push ( DOMString  cond_val,
DOMString  left_val,
DOMString  operand,
DOMString  right_val 
)

pushes a new condition instruction on the stack

Parameters
cond_valthe name of the condition. Allowed values are if, elseif
left_valthe name of the left value to be assigned or modified
operandthe name of the operand to use
right_valthe name of the right value to use by the operand
Returns
the 1-based index of the added instruction

◆ push() [4/6]

long Shader::push ( DOMString  end_cond_val)

pushes an end of condition instruction on the stack

Parameters
end_cond_valthe name of the end of condition. Allowed values are else, end
Returns
the 1-based index of the added instruction

◆ push() [5/6]

long Shader::push ( DOMString  goto_val,
long  stack_index 
)

pushes a goto instruction on the stack

Parameters
goto_valthe name of the goto. Allowed values are goto
stack_indexthe 1-based index of the stack instruction to go to
Returns
the 1-based index of the added instruction

◆ push() [6/6]

long Shader::push ( DOMString  goto_val,
DOMString  stack_index_uniform 
)

pushes a goto instruction on the stack

Parameters
goto_valthe name of the goto. Allowed values are goto
stack_index_uniformname if the uniform giving the 1-based index of the stack instruction to go to
Returns
the 1-based index of the added instruction

◆ update()

void Shader::update ( )

updates uniforms in the shader

GF_EVG_SHADER_FRAGMENT
@ GF_EVG_SHADER_FRAGMENT
Definition: evg.idl:1420
GF_EVG_SHADER_VERTEX
@ GF_EVG_SHADER_VERTEX
Definition: evg.idl:1422
GF_EVG_VAI_VERTEX
@ GF_EVG_VAI_VERTEX
Definition: evg.idl:1431
GF_EVG_VAI_PRIMITIVE
@ GF_EVG_VAI_PRIMITIVE
Definition: evg.idl:1433