Documentation update

This is update of documentation draft before code implementation.

13 files changed, 211 insertions, 94 deletions

diff --git a/.gitignore b/.gitignore
index de0941f..901d796 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,4 +2,11 @@
 .*
 !/.gitignore
 
+build
 sgp
+
+# Comment these for release. They are generated files using gperf and bison. For
+# final release they can be in repository. But during development ignore them.
+*.y.c
+*.y.h
+*.gperf.h
diff --git a/README.md b/README.md
index 8a81c5b..e0f8be5 100644
--- a/README.md
+++ b/README.md
@@ -1,2 +1,8 @@
 Small General Preprocessor
 ==========================
+This is intended as simple text preprocessor with possible rich feature set for
+expanding its capability.
+
+This tool can help you write text, may it be code, website or plain text, with
+special code embedded. Preprocessor than resolves all code and result is given to
+you. This way you can get unlimited number of different results from single file.
diff --git a/doc/syntax.md b/doc/syntax.md
deleted file mode 100644
index de9e725..0000000
--- a/doc/syntax.md
+++ /dev/null
@@ -1,94 +0,0 @@
-Syntax
-------
-Syntax consist from commands and variables. Both are defined in code with
-initial character. In default `$`, but others can be used if specified as
-argument to program. After this character is name of variable or command. Names
-are case sensitive, without spaces and special character.
-
-If you want use initial character without preprocessing, you should write it
-two times. So if our initial character is `$`, you should use `$$`.
-Also you can use `--permissive` argument to disable error when initial character is
-not followed by known name. In such case is preprocessor threating it as it is
-part of code and let it be without change. Be aware that collisions with names can
-occur and no typing errors will be reported.
-
-### Variables
-Only variables preprocessor knows are text based. So even if some calculation has
-to be made it is made with text. This slows down calculations, but removes
-requirement of types definition. Special case are arrays.
-
-If you use variable in file, it will be replaced directly with its value. So for
-example if there would be defined variable "VAR" with value "Hello", result after
-preprocessing of line
-```
-$VAR World!
-```
-would be
-```
-Hello World!
-```
-
-To set variable, see "set" command. To do mathematical operations see "eval"
-command.
-
-#### Arrays
-Arrays are variables containing multiple texts. This can be handy when you want
-store unknown number of text strings. Arrays in this preprocessor can have only
-one dimension. They are created by command "set" and can be walked through with
-command "for". Command "size" returns number of elements in array. Command "unset"
-is removing elements from array.
-
-Elements in array are indexed. Accessing some value is done with square bracket
-and number. For example value of 0 index of array with name "ARRAY" would be
-accessed with this syntax: "$ARRAY[0]". If array is referenced without brackets,
-only first value is used. So value of index 0 is also accessible with "$ARRAY".
-
-#### Special variables
-Some variable names are reserved. Such variables can be used for various purposes.
-##### SGP_VERSION
-Contains version of sgp tool.
-##### FILE
-Array with names of all files that preprocessor is now working with. They are
-ordered so current file is on index 0 and file including this file is on index 1
-and so on.
-##### LINE
-Array with numbers of lines where preprocessor is now reading. They are ordered so
-current line of current file is on index 0 and line in file including current file
-is on index 1 and so on.
-
-### Commands
-Every command consist of its name and possible arguments. Arguments are enclosed
-in brackets and between name and brackets are no other characters. Every argument
-can be specified on multiple lines and separated from others with comma. Because
-all evaluation is text based, arguments are also threated as text. Every empty
-character (null, new line, space, tabulator) is removed from beginning and end of
-argument. Also `"` and `'` characters are removed from beginning and end of
-argument, but no other character is removed from beginning after them and after
-last one on end should be only empty character. This allows common string
-enclosures. Also if command has fixed number of arguments, last argument can
-contains comas without being enclosured with `"` or `'`.
-
-Some commands can be paired with some other commands. Common example is "if" with
-its paired commands "else", "elif" and "endif"
-##### include
-##### set
-##### unset
-##### eval
-##### if
-##### ifdef
-##### ifndef
-##### if*
-##### elif
-##### else
-##### endif
-##### for
-##### endfor
-##### while
-##### endwhile
-##### exec
-##### error
-##### warning
-##### define
-##### undefine
-
-#### Defining more commands
diff --git a/docs/builtin_commands.md b/docs/builtin_commands.md
new file mode 100644
index 0000000..79222ad
--- /dev/null
+++ b/docs/builtin_commands.md
@@ -0,0 +1,107 @@
+Built-in commands
+=================
+Here is full list of all build in commands including its description and sometime
+usage examples. For more information see syntax documentation.
+
+All build in commands are defined in lover case as they are presented here, but
+also in upper case. Also you can change initial character, we use here default one
+(`$`).
+
+###	$include(PATH)
+Includes file content to its place. `PATH` is system path relative to current file
+folder or absolute path.
+
+With this command you can nest any other file content and this content. Note that
+because of evaluation principles, content can be evaluated multiple times. If it
+is not what you want, common idiom would be use:
+```
+$limit(0)$include(file)$endlimit
+```
+
+### $load(...)
+This loads additional modules. sgp searches with precedence for modules in file
+directory and in paths defined by environment variable `SGPMODULES`. `NAME` is in
+such case relative path to module, but also absolute path can be used.
+
+Be aware that modules stays loaded even after current file preprocessing is
+finished. They are not unloaded automatically after return from include.
+
+For more information see module documentation.
+
+### $unload(...)
+Serves as opposite to `load` command. It is as it says unloading modules. But it
+is not paired with load.
+
+Without any argument `unload` will unload all loaded modules.
+
+### $limit(NUM)
+This command limits number of evaluation cycles for some text (see evaluation
+documentation).  After this command all text will be evaluated only `NUM` times.
+So `NUM` should be number. Passing empty string results in disabled evaluation
+until end of limit command. Default value if not limit command is used is `inf`,
+that means no limit.  Limit can be ended using `endlimit` command.
+
+Because of how limit works. Most likely you will use limit without any argument or
+with `inf`, or with `1`. Lets see once again what every one does. No argument is
+same as `0` so no evaluation is done until `endlimit` command. `inf` removes any
+limit and allows any number of nested evaluations. `1` specifies that evaluation
+should be done only once.
+
+You can nest limits together. This results to situation when content is evaluated
+limited times but some part of it that is wrapped in another limit block will be
+evaluated few times more without sucking up its root limit.
+
+### $endlimit
+This ends previous evaluation limit. Text after this command is not limited by
+that.
+
+### $"
+### $'
+This is almost same command as `$"`. It has same behavior as if you would write:
+```
+$"$limit $Text $endlimit$"
+```
+Where result would be:
+```
+ $Text 
+```
+So its also limits evaluation.
+
+### $set(VAR,VALUE)
+### $define(VAR,...)
+### $enddefine
+### $undefine(VAR)
+### $math(EQ)
+### $if(COND)
+### $ifdef(VAR)
+### $ifndef(VAR)
+### $elif(COND)
+### $else
+### $endif
+### $for(VAR,COND,EQ)
+### $endfor
+### $while(COND)
+### $endwhile
+### $switch(VAR)
+### $case(VAL)
+### $default
+### $break
+### $continue
+### $exec(EXEC,...)
+Calls executable `EXEC` with rest of arguments as arguments for executed program.
+Standard output from this program is than included to same place as `exec`.
+
+### $shell(CMD)
+This is same as exec, except that instead of defining executable and command line
+arguments separately they are defined together in `CMD` argument. In fact you can
+specify any shell command in `CMD`.
+
+### $error(TEXT)
+`TEXT` is printed to error output and spg is terminated.
+
+### $warning(TEXT)
+`TEXT` is printed to error output, but sgp continuous preprocessing text.
+
+### $info(TEXT)
+Simply prints `TEXT` to standard output. If standard output is used as output of
+result, info is printed to error output.
diff --git a/docs/evaluation.md b/docs/evaluation.md
new file mode 100644
index 0000000..04d8854
--- /dev/null
+++ b/docs/evaluation.md
@@ -0,0 +1,2 @@
+Evaluation process
+==================
diff --git a/docs/index.md b/docs/index.md
new file mode 120000
index 0000000..32d46ee
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1 @@
+../README.md
\ No newline at end of file
diff --git a/doc/command.md b/docs/module_creation.md
index e69de29..e69de29 100644
--- a/doc/command.md
+++ b/docs/module_creation.md
diff --git a/docs/modules/array.md b/docs/modules/array.md
new file mode 120000
index 0000000..1c9ed09
--- /dev/null
+++ b/docs/modules/array.md
@@ -0,0 +1 @@
+../../modules/array/README.md
\ No newline at end of file
diff --git a/docs/modules/file.md b/docs/modules/file.md
new file mode 120000
index 0000000..82313a3
--- /dev/null
+++ b/docs/modules/file.md
@@ -0,0 +1 @@
+../../modules/file/README.md
\ No newline at end of file
diff --git a/docs/modules/protect.md b/docs/modules/protect.md
new file mode 120000
index 0000000..5a63a23
--- /dev/null
+++ b/docs/modules/protect.md
@@ -0,0 +1 @@
+../../modules/protect/README.md
\ No newline at end of file
diff --git a/docs/modules/python.md b/docs/modules/python.md
new file mode 120000
index 0000000..cdce586
--- /dev/null
+++ b/docs/modules/python.md
@@ -0,0 +1 @@
+../../modules/python/README.md
\ No newline at end of file
diff --git a/docs/syntax.md b/docs/syntax.md
new file mode 100644
index 0000000..fa00287
--- /dev/null
+++ b/docs/syntax.md
@@ -0,0 +1,66 @@
+Syntax
+======
+Syntax consist from expandable macros and commands. They are both used in code
+with initial character. In default `$`, but other can be used if specified as
+argument to program. After this character is name of macro. Names are case
+sensitive, without spaces and special character, except underscore.
+
+For example this text:
+```
+$set(VAR, Hello)$VAR World!
+```
+Is expanded to:
+```
+Hello World!
+```
+
+If you want use initial character without preprocessing, you should write it two
+times. So if our initial character is `$`, you should use `$$`.  Also you can use
+`--permissive` argument to disable error when initial character is not followed by
+known name. In such case is preprocessor treating it as it is part of text and let
+it be without change. Be aware that collisions with names can occur and no typing
+errors will be reported. Also you can limit preprocessing with `limit` command. If
+you set limit, only given number of macros and commands is expanded.
+
+Macros and commands can have arguments. Arguments are used to pass text to them.
+Every argument can be specified on multiple lines and separated from others using
+coma. All empty characters (spaces, tabulators and new lines) are removed from
+beginning and end of argument. You can enclosure argument using `$'` or `$"`, which
+will ensure that empty characters will stay in argument. But be aware that even
+when `$'` or `$"` is used macros and functions inside argument is evaluated. Another
+usage for enclosing argument to `$'` or `$"` is to ignore `)` and `,`.
+
+End of macro or command is identified by other character that is not known. Or
+bracket that is used to bound arguments to commands and macros. Or you can use `-`
+as special meaning. `-` is removed during processing which allows you to paste
+anything anywhere. For example `Without$SPACE-spaces`.
+
+Difference between macros and commands is that macros are defined in text and
+allows only simple expansion. Additional commands are defined using modules.
+At the end, commands are more versatile than macros, but requires more work to
+create.
+
+Some commands can be paired with some other commands. Common example is "if" with
+its paired commands "else", "elif" and "endif". Paired commands are defined as
+that first of them defines behavior and second one ends it or changes it for next
+text. It should newer be that second one somehow defines behavior for previous
+processed text. Also to ensure that code is contained inside file, all first
+commands allowing pairs has default ending command. If file is read to end and no
+pair is found, default end is artificially inserted. Also no paired command opened
+before including can be ended in included file.
+
+Special macros
+--------------
+These macros has special meaning. They can't be redefined, but can be overloaded
+using modules. And they are not defined in processed text.
+### SGP_VERSION
+Expands to version of sgp tool.
+### FILE
+Expands to name of file that is processed. This macro is not changing during whole
+preprocessing. Note that you can use it in included files to identify name of
+original preprocessed file.
+### LINE
+This macro is expanded always to line number on which it is specified in source
+text.
+### MODULES
+This macro specifies coma separated list of active modules.
diff --git a/mkdocs.yml b/mkdocs.yml
new file mode 100644
index 0000000..f8037d6
--- /dev/null
+++ b/mkdocs.yml
@@ -0,0 +1,18 @@
+site_name: SGP
+repo_url: https://github.com/Cynerd/sgp.git
+site_description: Simple genetal preprocessor
+copyright: GNU General Public License, version 2.0
+pages:
+- Home: index.md
+- Syntax: syntax.md
+- Built-in commands: builtin_commands.md
+- Evaluation process: evaluation.md
+- Modules:
+    - Array: modules/array.md
+    - File: modules/file.md
+    - Protect: modules/protect.md
+    - Python: modules/python.md
+- Modules creation: module_creation.md
+theme: readthedocs
+
+site_dir: html