Start fleshing out documentation

README.md

@@ -1,38 +1,114 @@
-Point Cloud
-===========
+Pointcloud
+==========
 
 A PostgreSQL extension for storing point cloud (LIDAR) data.
 
-Requires
-========
-
-- PostgreSQL 9.1+ (support for extensions)
-- PostgreSQL compiled --with-xml
-
 
 Build/Install
 =============
 
-- Requires PostgreSQL dev packages (pg_config and server headers), usually 'postgresql-dev' or 'postgresql-devel'
-- Edit config.mk and update paths to libxml2 and cunit
-- make
-- sudo make install
-- create a database
-- run 'CREATE EXTENSION pointcloud'
+Requirements
+------------
 
-Formats
+- PostgreSQL and PostgreSQL development packages must be installed (pg_config and server headers). For Red Hat and Ubuntu, the package names are usually "postgresql-dev" or "postgresql-devel"
+- LibXML2 development packages must be installed, usually "libxml2-dev" or "libxml2-devel".
+- CUnit packages must be installed, or [source built and installed](http://sourceforge.net/projects/cunit/ "CUnit").
+
+Build
+-----
+
+- `./configure`
+- `make`
+- `sudo make install`
+
+Activate
+--------
+
+- `CREATE DATABASE mynewdb`
+- `CREATE EXTENSION pointcloud`
+
+
+Schemas
 =======
 
-In order to preserve some compactness in dump files and network transmissions, the binary formats need to retain their native compression. The human readable format is mostly for debugging and tests, and is not compressed. Uncompressed binary is, as advertised, just hex-encoded points and sets of points.
+LIDAR sensors quickly produce millions of points with large numbers of variables measured on each point. The challenge for a point cloud database extension is efficiently storing this data while allowing high fidelity access to the many variables stored. 
 
-Human-Readable Text
--------------------
+Much of the complexity in handling LIDAR comes from the need to deal with multiple variables per point. The variables captured by LIDAR sensors varies by sensor and capture process. Some data sets might contain only X/Y/Z values. Others will contain dozens of variables: X, Y, Z; intensity and return number; red, green, and blue values; return times; and many more. There is no consistency in how variables are stored: intensity might be stored in a 4-byte integer, or in a single byte; X/Y/Z might be doubles, or they might be scaled 4-byte integers. 
 
-    ( <pcid> : <dim1>, <dim2>, <dim3>, <dim4> )
+PostgreSQL Pointcloud deals with all this variability by using a "schema document" to describe the contents of any particular LIDAR point. Each point contains a number of dimensions, and each dimension can be of any data type, with scaling and/or offsets applied to move between the actual value and the value stored in the database. The schema document format used by PostgreSQL Pointcloud is the same one used by the [PDAL](http://pointcloud.org) library.
 
-    [ <pcid> : ( <dim1>, <dim2>, <dim3>, <dim4> ), 
-               ( <dim1>, <dim2>, <dim3>, <dim4> ), 
-               ( <dim1>, <dim2>, <dim3>, <dim4> ) ]
+Schema documents are stored in the `pointcloud_formats` table, along with a `pcid` or "pointcloud identifier". Rather than store the whole schema information with each database object, each object just has a `pcid`, which serves as a key to find the schema in `pointcloud_formats`.  This is similar to the way the `srid` is resolved for spatial reference system support in [PostGIS](http://postgis.net).
+
+The central role of the schema document in interpreting the contents of a point cloud object means that care must be taken to ensure that the right `pcid` reference is being used in objects, and that it references a valid schema document in the `pointcloud_formats` table.
+
+
+Point Cloud Objects
+===================
+
+PcPoint
+-------
+
+The basic point cloud type is a `PcPoint`. Every point has a (large?) number of dimensions, but at a minimum an X and Y coordinate that place it in space. 
+
+Points can be rendered in a human-readable JSON form using the `PC_AsText(pcpoint)` function. The "pcid" is the foreign key reference to the `pointcloud_formats` table, where the meaning of each dimension in the "pt" array of doubles is explained. The underlying storage of the data might not be double, but by the time it has been extracted, scaled and offset, it is representable as doubles.
+
+    {
+        "pcid" : 1,
+          "pt" : [0.01, 0.02, 0.03, 4]
+    }
+
+PcPatch
+-------
+
+The structure of database storage is such that storing billions of points as individual records in a table is not an efficient use of resources. Instead, we collect a group of `PcPoint` into a `PcPatch`. Each patch should hopefully contain points that are near together. 
+
+Instead of a table of billions of single `PcPoint` records, a collection of LIDAR data can be represented in the database as a much smaller collection (10s of millions) of `PcPatch` records. 
+
+Patches can be rendered into a human-readable JSON form using the `PC_AsText(pcpatch)` function.  The "pcid" is the foreign key reference to the `pointcloud_formats` table.
+
+    {
+        "pcid" : 1,
+         "pts" : [
+                  [0.02, 0.03, 0.05, 6],
+                  [0.02, 0.03, 0.05, 8]
+                 ]
+    }
+
+
+Functions
+=========
+
+**PC_MakePoint(pcid integer, vals float8[])** returns **pcpoint**
+
+> Given a valid schema `pcid` and an array of doubles that matches the schema, construct a new PcPoint.
+> 
+>     SELECT PC_MakePoint(1, ARRAY[-127, 45, 124.0, 4.0]);
+
+    
+
+Binary Formats
+==============
+
+In order to preserve some compactness in dump files and network transmissions, the binary formats need to retain their native compression.  All binary formats are hex-encoded before output. 
+
+The point and patch binary formats start with a common header, which provides:
+
+- endianness flag, to allow portability between architectures
+- pcid number, to look up the schema information in the `pointcloud_formats` table
+
+The patch binary formats have additional standard header information:
+
+- the compression number, which indicates how to interpret the data
+- the number of points in the patch
+
+After the header comes the data. Data are packed into the data area following the data types and sizes 
+
+
+There are two compression schemes currently implemented
+
+- 0, uncompressed, is a simple set of points
+
+), Uncompressed binary is, as advertised, just hex-encoded points and sets of points.
 
 Point Binary
 ------------

pgsql/pointcloud_postgis--1.0.sql

@@ -16,24 +16,42 @@ CREATE OR REPLACE FUNCTION PC_Intersection(pcpatch, geometry)
 -----------------------------------------------------------------------------
 -- Cast from pcpatch to polygon
 --
-CREATE OR REPLACE FUNCTION pcpatch2geometry(pcpatch)
+CREATE OR REPLACE FUNCTION geometry(pcpatch)
     RETURNS geometry AS
     $$
         SELECT ST_GeomFromEWKB(PC_Envelope($1))
     $$ 
     LANGUAGE 'sql';
 
-CREATE CAST (pcpatch AS geometry) WITH FUNCTION pcpatch2geometry(pcpatch);
+CREATE CAST (pcpatch AS geometry) WITH FUNCTION geometry(pcpatch);
 
 -----------------------------------------------------------------------------
 -- Cast from pcpoint to point
 --
-CREATE OR REPLACE FUNCTION pcpoint2geometry(pcpoint)
+CREATE OR REPLACE FUNCTION geometry(pcpoint)
     RETURNS geometry AS
     $$
         SELECT ST_GeomFromEWKB(PC_AsBinary($1))
     $$ 
     LANGUAGE 'sql';
 
-CREATE CAST (pcpoint AS geometry) WITH FUNCTION pcpoint2geometry(pcpoint);
+CREATE CAST (pcpoint AS geometry) WITH FUNCTION geometry(pcpoint);
+
+
+-----------------------------------------------------------------------------
+-- Function to overlap polygon on patch
+--
+CREATE OR REPLACE FUNCTION PC_Intersects(pcpatch, geometry)
+    RETURNS pcpatch AS
+    $$
+        SELECT ST_Intersects($2, geometry($1))
+    $$ 
+    LANGUAGE 'sql';
+
+CREATE OR REPLACE FUNCTION PC_Intersects(geometry, pcpatch)
+    RETURNS pcpatch AS
+    $$
+        SELECT PC_Intersects($2, $1)
+    $$ 
+    LANGUAGE 'sql';