Automatic operation

# NEURO_VJ

Some links will not be active if there is no internet connection.
BEWARE: this application can only run on anyflo version 16.2 jully 2014.

## Principle
The menus
Programmable effects
Keyboard Shortcuts
Examples
Automatic operation
The program
Launching options
Images converter

# Principe

A Dynamic Analysis of the fft audio signal (or its derivative) provides real-time, to each image, an M1 parameter list that is passed as input (pattern) of a neural network another L1 (law) parameters that are interpreted in terms of dynamic volumes and textures.
A list M2of maximum modules fft increased input (pattern) of a second neural network which provides the output, another list L1 (law) parameters that are interpreted in terms of succession of sequences.
En fait c´est l´In fact it is the envelope de la fft of the fft being treate.
To assure processing in real a set of Image files, read at program startup time are interpolated to provide continuous transitions.
Networks are dynamically trained by algorithm back-propagation of the error, its entries being updated periodically update the audio signal. It follows an animation leaning back permanently on music. Different effects can be programmedeffets.

##
Note:
Message "Must press w key to start audio" appears.

# The menus

## HEL
EXP
INPUT
OUTPUT
NET1
NET2
AUTO
EFFECT
SCALES
RESET
ILLUM
EXT
EXI

## HEL

Help.

## EXP

### HEL
AUD
NET

### HEL

Help.

### AUD

HEL
RING
AMP
FFT
STOP
HEL
Help.
RING
Change freq (frequency) and dur (duration).
AMP
Change audio and amplitude.
FFT
Change audio and amplitude.
STOP
Stops sound.

### NET

HEL
BUILD
LEARN
MATRIX
HEL
Help.
BUILD
Building a neural network.
Change couples=number of training couples.
Change hide=number of hidden layers.
LEARN
Learning the neural network, produce sound.
MATRIX
Displays the matrix of synaptic weights.

## INPUT

### HEL
AUD
CAM

### HEL Help.

### AUD

Audio.

### CAM

Camera: The window is shifted to the right, leaving room for the window it took WebCam.exe launch earlier stages, webcam well as a camp whose image properly resized, has been brought on the first, otherwise. the program crashes ...
Change scales:
liminance rgb: luminance (1D) or RGB (3D).
dim cam: size of the camera image displayed..
secx secy: size of the captured image, also dimension of patterns and laws of network that will rebuild.
speed: rate of change patterns.
Then just click on AUTO to start the process.

## OUTPUT

### HEL
INI
IMA
CAM
CAM2

### HEL

Aide.

### INI: ini image

### IMA

HEL
READ
DIRECTORY
BUILD
PIXEL
Ces différents modes peuvent être combinés.
HEL
Help.
READ
reads images: click on a directory name, hen on a name.bmp.
DIRECTORY
lit reads images folders: click on a directory name.
BUILD
builds random images, click on nb = number of images and on complex = complexity;
PIXEL
builds images curves, click on nb = complexity;

### CAM

Web cam captures an image and displays transparent texture (must be launched webcam.js and have launched a web cam and bringing its image (appropriately scaled) on the previous window, then click click here.

### CAM2

As CAM2, but the image is captured only when the transparency of the last layer is null.

## NET1

Neural network transforming the fft of the audio sugnal. Change couples (number of learning couples); and
hide (number of hidden layers).

### HEL
INI
LEARN
MATRIS

### HEL

Aide.

### INI

Removes the network.

### LEARN

Launches the learning process.
On peut see the input layer neurons (green), those of the ouptut layer (red) as well as those of the hidden layers activate.

### MATRIX

Displays the matrix of synaptic weights.

## NET2

Neural network transforming lelory Memroy into values passed to interaction image(1,Nb_Seq)vol law(m); in order to determin the following sequence. Change:
couples (number of learning couples); and
hide (number of hidden layers).

### HEL
INI
LEARN
MATRIS

### HEL

Help.

### INI

Supprime le réseau.

### LEARN

Launches the learning process.
We can see the input layer neurons (green), those of the output layetr (red) as well as those of the hidden layers activate.
On the right we can see the error curve (white).

### MATRIX

displays the matrix of synaptic weights.

## AUTO

Launches the process.
Click on reset change the refresh period of the network.
Click on ni change the of interpolated images interpolées, this may take some time.

# Programable effects

### HEL
SELF
PREC
LIGHT
COL
VIEW
VOL
SIN
net
DEB
DER
CONT
MOVE
BACK
NOISE
NEXT
ALEA
LAW
TRA
AMP
CUR
TAB
WAIT

### HEL

Help.

### SELF

Activates the automapping (keyboard shortcut s).

### SHOW

Displays the audio signal, the motifs and the laws of both networks (even if : is active).

### PREC

### image precision sends a thresholding of the image.

### LIGHT

Computes the color of light depending on the average fft (keyboard shortcut l).

### COL

Computes the color of volume depending on the average fft, change the scales coe COL and pow (a small value of pow improves contrasts.

### VIEW

Variable point of view (keyboard shortcut v).

### VOL

Volume trajectories (keyboard shortcut V).

### SIN

Sinusoidal variationde of the texture increment.

### net

Transforms (or not) the audio signal audio by the network Net1 (keyboard shortcut n).

### DEB

Debug mode: prints some parameters for the launch(heyboard shortcut d).

### DER

Does Der[2]=1 (sensor speed) (keyboard shortcut D).

### CONT

Contour extraction.

### MOVE

Fixed points of the grid (their boundaries) are moved according to the fft, change the scale coe MOVE (keyboard shortcut M).

### BACK

The background color varies (keyboard short shortcut b).

### NOISE

The forces applied to the vertices are noised of coe NOISE, available on SCALES VOL (keyboard shortcut Z).

### NEXT

Following sequence.

### ALEA

Random sequences.

### LAW

Sequences determined by the maximum value of the memory Memory by the network Net2 (keyboard shortcut L).

### AMP

Sequences determined by Module*max(number image). (keyboard shortcut a).

### CUR

sequence determined by law(Module,Curve) (keyboard shortcut k).
To change the curve Curve click on SCALES -> PARAM, change np, then the curve.

### TRA

Images are mixed by image(i1,i2)center(x,y)radius(rx,ry)transp(t1,t2)normal

### TAB

The look up table is changed according to the values of the module stocked in the FILO stack Memory.

### WAIT

Audio OFF/ON (ON defautl: must press w key to start audio.

## SCALES

### HEL
OUTPUT
AUD
CAM
VOL
PARAM

### HEL

Help.

### OUTPUT

dim: images.
dim: image size.
r,g,b: coefficients thresholds color if PREC is active.

### AUD

audio: number of samples= size of the input motifs = 2^audio. It is then necessary to click on AUTO.
envelope: envelope.
above: if validates audio var sort above (signal reduced to the maximum component).
derive: capture = speed.
min: minimum of the audio signal, allows ignore too low values.
maw: maximum of the audio signal, allows ignore too hight values.
amp: multiplier signal.
exp,adjust,rand: signal adjustement.
per: refresh period motifs;
speed: textures speed variation.
coe audio=c:
if c >0: does coe audio=c.
if <0: choose m1 and m2 making coe audio=m1,m2 (automatic coefficient).

### CAM

luminance rgb: luminance or RVG.
dim cam: image (power of 2).
secx secy: camera resolution. resolution

### VOL

texture: number of textures.
If CONT is validated:
pricision and smooth
stiffness,coe stiffness and viscosity of the springs..
max maximum textures increment (avoids swings).
viscosity: viscosity.
hide: number of hidden layers.
ni: determines the size of the image interpolation.
sin1,sin2: if SIN is va lidated changes the sinusoidalvariations of the textures increment.

### PARAM

If href="#TAB">TAB

is active, displays thelook up table curve.
If raw is activedisplay the name of the image file currently processed and the scale coe period. The files must be in format raw.
If option CUR is active, displays np and the curve Curve thar can be changed.

## RESET

Restores original (afetr, for example, SELF or PREC).

## ILLUM

yes/no illum.

## EXT

yes/no ext, change the scaleext.

## INI

Restores the original parameters.

## EXI

Exits.

# Raccourcis clavier

: (colon) hides menus.
A: random suquences (otherwise linear) pour
m mini menu appears in the upper left with options:
A ALEA
b BACK
c COL
C CONT
D DER
d DEB
h help mini menu (keyboard shortcut).
i initialisation
l LIGHT
L LAW
m mini menu
M MOVE
n NET
N NEXT
p PREC
r RESET
s SELF
S SHOW
t TRA
T TAB
v VIEW
V VOL
Z NOISE
: cache les menus
+ incrments amp
- decrements amp
q EXI

# Examples

### Direct
Manual

### Direct

Click on neuro_VJ_raw.js launching neuro_VJ.func with option raw which dynamically read images format raw of directory RAW.
Click on neuro_VJ_bmp.js launching neuro_VJ.func with option dir=ima/BMP which reads images format bmp of directory BMP.
Click on webcam.js launching a shared memory driver.
Then launche a webcam, drag the resized image on the window of the driver (lower left corner).
Then click on neuro_VJ_cam.js launching neuro_VJ.func with options:
cam readin the captured image.
and dir=ima/BMP which reads images format bmp of directory BMP.
Play music and act front of the camera.

### Manual

Lancer neuro_VJ.js without option.
Click on INPUT then on AUD, making audio the input sensor.
Click on OUTPUT then on IMA then on DIRECTORY and select folders>, this starts the process: all images of selected folders are loaded and used as interpolated transparency textures.
Click on AUTO
Press : (colon) key hidding menus.
Press m key giving mini menu.
Pressh key giving mini menu help.
Press documented keys to make several effects.
Press : key gives menus.
Click on SCALES, then,
OUTPUT.
AUD.
VOL.
NET1.
NET2.

# Automatic operation

Activate NET1, NETWORK->LEARN, and AUTO.
Note: if INPUT_AUD is validated, images must have been read.

# The program

### The function

The function neuro_VJ.func is in the folder ima.

### The sequences

If net is not validated the signal is directly interpreted (the message DIRECT is displayed above).
Otherwise the signal is passed in entrance of the neural network Net1 which gives as output the interpreted signal. It is necessary to have read the images or to have specified an option dir.
The option dir=dir_1,dir_2,...,dir_n produces succession of sequences according to various criteria:
random (keyboard shortcut A).
next (keyboard shortcut N).
law (keyboard shortcut L).
amplitude (keyboard shortcut a).
curve (keyboard shortcut k).
One can build as many as such directories as needed. But, as the images are loaded in memory, the number of images they contain is limited according to memory size. In that case the option raw allows dynamically read as many images as needed. For that it will be necessary to launch bmp_raw.js.

### Volumes

A grid is buildt with:
secx = secy = Nxy = dim(Capture)
memory(1) = numbers of summits not subjected to the dynamic (but directly modified by the fft.
memory(2) = coordinates of those points.
ver=memory(1)vol(num);n=dim(ver);p=memory(2)vol(num);
(in function func_AUTO)
`memory("a")vol(num)=(average(Capture))^e;`

# Launching options

aud=n: does Nb_AUDIO[2]=n (9 default).
aut: activates the automatic operation automatique.
cam: activates the camera.
cont=p: contour of precision p (0.025 default).
cur=0,0,0, x1,y1,0, x2,y2,0, ..., 1,1,0: Curve.
der: doest Derive[2]=1 (sensor speed).
dim=d: size of the images=2^d (7 default).
dir=ima/BMP: reads images of folder BMP.
dir0=dos: reads folders of folder dos.
env=n: does Envelope[2]=n (6 default).
1: gives an example of programmable option, see case 1: of function option(w) reading two directories BMP1,BMP2. File neuro_VJ_1.js uses such an option.
raw: only active images are loaded allowing interpolations between an unlimited number of images with arbitrariy size (butin this case, option TAB does not work in real time). The interpolation of textures is made by transparency with properties:
texture vol(1)=t1,t2,... and
transp texture vol(1)=tr1,tr2,...).
The array Transp = {(transp,signe,0)i of n 3D vectors 3D allows managing the succession of sequences.
texture=nb: number of textures.
Example:
quot;dir=ima/BMP1,ima/BMP2;
reads images of folders ima/CAH_1 and ima/SUZ_1
ni=n: number of intercalated images between 2 consecutive images (12 default).
:: hides menus.
a: random succession of images (otherwise linear).
b: activates the background color.
c: activates the colors.
l: activates the light.
p: activates limage precision.
q: inhibits the q key (avoiding existing).
s: activates the selfomapping.
t: activates transparency.
T: activates the look up table.
v: activates function VIEW.
V: activates function VOL.
Z: activates function NOISE with coefficient c.
Z: does WAIT OFF.
neuro_VJ.js: launch without option.
neuro_VJ_cam.js: option cam
neuro_VJ_cam2.js: option cam2
neuro_VJ_raw.js: option raw
Note:
If at least one option is precised (automatic operation) the option EXI is inhibited, to exit must:
: retriving menus.
and click on EXI.

# Images converter

File jpg_bmp.js launches file jpg_bmp.func allowing converting images from ima/JPG/*.jpg of any size in ima/BMP/*.bmp of size 256.
File bmp_raw.js launches file bmp_raw.func allowing converting images from ima/BMP/*.bmp of size 256 in ima/RAW_2/*.raw.