Documentation

README.md

@@ -124,9 +124,10 @@ Patches can be rendered into a human-readable JSON form using the `PC_AsText(pcp
 
 Usually you will only be creating tables for storing `PcPatch` objects, and using `PcPoint` objects as transitional objects for filtering, but it is possible to create tables of both types. `PcPatch` and `PcPoint` columns both require an argument that indicate the `pcid` that will be used to interpret the column. 
 
+    ```sql
     -- This example requires the schema entry from the previous 
     -- section to be loaded so that pcid==1 exists.
-    
+
     -- A table of points
     CREATE TABLE points (
         id SERIAL PRIMARY KEY,
@@ -137,7 +138,7 @@ Usually you will only be creating tables for storing `PcPatch` objects, and usin
     CREATE TABLE patches (
         id SERIAL PRIMARY KEY,
         pa PCPATCH(1)
-    );
+    );```
 
 In addition to any tables you create, you will find two other system-provided point cloud tables,
 
@@ -294,6 +295,81 @@ Now that you have created two tables, you'll see entries for them in the `pointc
 >      {"pcid":1,"pt":[-126.92,45.08,8,0]} |  1
 >      {"pcid":1,"pt":[-126.91,45.09,9,0]} |  1
 
+**PC_PatchAvg(p pcpatch, dimname text)** returns **numeric**
+
+> Reads the values of the requested dimension for all points in the patch 
+> and returns the *average* of those values. Dimension name must exist in the schema.
+>
+>     SELECT PC_PatchAvg(pa, 'intensity') 
+>     FROM patches WHERE id = 7;
+>
+>     5.0000000000000000
+
+**PC_PatchMax(p pcpatch, dimname text)** returns **numeric**
+
+> Reads the values of the requested dimension for all points in the patch 
+> and returns the *maximum* of those values. Dimension name must exist in the schema.
+>
+>     SELECT PC_PatchMax(pa, 'x') 
+>     FROM patches WHERE id = 7;
+>
+>     -126.41
+
+**PC_PatchMin(p pcpatch, dimname text)** returns **numeric**
+
+> Reads the values of the requested dimension for all points in the patch 
+> and returns the *minimum* of those values. Dimension name must exist in the schema.
+>
+>     SELECT PC_PatchMin(pa, 'y') 
+>     FROM patches WHERE id = 7;
+>
+>     45.5
+
+## Compressions ##
+
+One of the issues with LIDAR data is that there is a lot of it. To deal with data volumes, PostgreSQL Pointcloud allows schemas to declare their preferred compression method in the `<pc:metadata>` block of the schema document. In the example schema, we declared our compression as follows:
+
+    ```xml
+    <pc:metadata>
+      <Metadata name="compression">dimensional</Metadata>
+    </pc:metadata>```
+
+There are currently two supported compressions:
+
+- "**None**" which stores points and patches as byte arrays using the type and formats described in the schema document.
+- "**Dimensional**" which stores points the same as 'none' but stores patches as collections of dimensional data arrays, with with an "appropriate" compression applied. Dimensional compression makes the most sense for smaller patch sizes, since small patches will tend to have more homogeneous dimensions.
+
+A third compression is in development:
+
+- "**Geohashtree**" or GHT which stores the points in a tree where each node stores the common values shared by all nodes below. For larger patch sizes, GHT should provide effective compression and performance for patch-wise operations.
+
+### Dimensional Compression ###
+
+Dimensional compression first flips the patch representation from a list of N points containing M dimension values to a list of M dimensions each containing N values.
+
+    "pcid":1,"pts":[
+          [-126.99,45.01,1,0],[-126.98,45.02,2,0],[-126.97,45.03,3,0],
+          [-126.96,45.04,4,0],[-126.95,45.05,5,0],[-126.94,45.06,6,0]
+         ]}
+
+Becomes
+
+    "pcid":1,"dims":[
+          [-126.99,-126.98,-126.97,-126.96,-126.95,-126.94],
+          [45.01,45.02,45.03,45.04,45.05,45.06],
+          [1,2,3,4,5,6],
+          [0,0,0,0,0,0]
+         ]}
+
+The potential benefit for compression is that each dimension has quick different distribution characteristics, and is amenable to different approaches.  In this example, the fourth dimension (intensity) can be very highly compressed with run-length encoding (one run of six zereos). The first and second dimensions have relatively low variability relative to their magnitude and can be compressed by removing the repeated bits.
+
+Dimensional compression currently supports only three compression schemes:
+
+- run-length encoding, for dimensions with low variability
+- common bits removal, for dimensions with variability in a narrow bit range
+- raw zlib compression, for dimensions that aren't amenable to the other schemes
+
+For LIDAR data organized into patches of points that sample similar areas, **...INSERT COMPRESSION RATIO ESTIMATE HERE AFTER EXPERIMENTATION..**
 
 
 ## Binary Formats ##
@@ -310,14 +386,6 @@ The patch binary formats have additional standard header information:
 - the compression number, which indicates how to interpret the data
 - the number of points in the patch
 
-After the header comes the data. Data are packed into the data area following the data types and sizes 
-
-
-There are two compression schemes currently implemented
-
-- 0, uncompressed, is a simple set of points
-
-), Uncompressed binary is, as advertised, just hex-encoded points and sets of points.
 
 ### Point Binary ###
 

lib/TODO.md

@@ -4,7 +4,5 @@ To Do
 - (?) convert PCBYTES to use PCDIMENSION* instead of holding all values as dupes
 - (??) convert PCBYTES handlign to pass-by-reference instead of pass-by-value
 
-- tests for dimensional compression in the database
-
-
-
+- implement PC_PatchAvg/PC_PatchMin/PC_PatchMax as C functions against patches with dimensional and uncompressed implementations
+- update pc_patch_from_patchlist to merge dimensional patchlists directly

pgsql/pc_access.c

@@ -18,6 +18,8 @@ Datum pcpatch_from_pcpatch_array(PG_FUNCTION_ARGS);
 Datum pcpatch_uncompress(PG_FUNCTION_ARGS);
 Datum pcpatch_numpoints(PG_FUNCTION_ARGS);
 Datum pcpatch_intersects(PG_FUNCTION_ARGS);
+Datum pcpatch_size(PG_FUNCTION_ARGS);
+Datum pcpoint_size(PG_FUNCTION_ARGS);
 
 /* Generic aggregation functions */
 Datum pointcloud_agg_transfn(PG_FUNCTION_ARGS);
@@ -549,3 +551,17 @@ Datum pcpatch_intersects(PG_FUNCTION_ARGS)
     PG_RETURN_BOOL(TRUE);
 }
 
+PG_FUNCTION_INFO_V1(pcpatch_size);
+Datum pcpatch_size(PG_FUNCTION_ARGS)
+{
+	SERIALIZED_PATCH *serpa = PG_GETARG_SERPATCH_P(0);
+    PG_RETURN_INT32(VARSIZE(serpa));
+}
+
+PG_FUNCTION_INFO_V1(pcpoint_size);
+Datum pcpoint_size(PG_FUNCTION_ARGS)
+{
+	SERIALIZED_POINT *serpt = PG_GETARG_SERPOINT_P(0);
+    PG_RETURN_INT32(VARSIZE(serpt));
+}
+

pgsql/pointcloud--1.0.sql

@@ -134,6 +134,13 @@ CREATE OR REPLACE FUNCTION PC_Intersects(p1 pcpatch, p2 pcpatch)
     RETURNS boolean AS 'MODULE_PATHNAME', 'pcpatch_intersects'
     LANGUAGE 'c' IMMUTABLE STRICT;
 
+CREATE OR REPLACE FUNCTION PC_MemSize(p pcpatch)
+    RETURNS int4 AS 'MODULE_PATHNAME', 'pcpatch_size'
+    LANGUAGE 'c' IMMUTABLE STRICT;
+
+CREATE OR REPLACE FUNCTION PC_MemSize(p pcpoint)
+    RETURNS int4 AS 'MODULE_PATHNAME', 'pcpoint_size'
+    LANGUAGE 'c' IMMUTABLE STRICT;
 
 -------------------------------------------------------------------
 --  POINTCLOUD_COLUMNS
@@ -278,13 +285,13 @@ CREATE OR REPLACE FUNCTION PC_PatchMax(p pcpatch, attr text)
     LANGUAGE 'sql';
 
 -- Utility to get MINIMUM value from patch
-CREATE OR REPLACE FUNCTION PC_PatchMax(p pcpatch, attr text)
+CREATE OR REPLACE FUNCTION PC_PatchMin(p pcpatch, attr text)
     RETURNS numeric AS
     $$
         WITH pts AS (
         SELECT PC_Explode(p) AS pt
         )
-        SELECT max(PC_Get(pt, attr)) FROM pts
+        SELECT min(PC_Get(pt, attr)) FROM pts
     $$ 
     LANGUAGE 'sql';
 

pgsql/sql/pointcloud.sql

@@ -129,15 +129,10 @@ INSERT INTO pa_test (pa) VALUES ('0000000001000000000000000200000006000000070000
 
 SELECT PC_AsText(pa) FROM pa_test;
 SELECT PC_Envelope(pa) from pa_test;
-
 SELECT PC_AsText(PC_Patch(pt)) FROM pt_test;
-
 SELECT PC_AsText(PC_Explode(PC_Patch(pt))) FROM pt_test;
-
 SELECT PC_AsText(PC_Union(pa)) FROM pa_test;
-
-SELECT sum(PC_NumPoints(pa)) FROM pa_test ;
-SELECT sum(PC_NumPoints(pa)) FROM pa_test ;
+SELECT sum(PC_NumPoints(pa)) FROM pa_test;
 
 CREATE TABLE IF NOT EXISTS pa_test_dim (
     pa PCPATCH(20)
@@ -148,6 +143,12 @@ INSERT INTO pa_test_dim (pa) VALUES ('000000001400000000000000020000000600000007
 INSERT INTO pa_test_dim (pa) VALUES ('0000000014000000000000000200000002000000030000000500060000000200000003000000050003');
 INSERT INTO pa_test_dim (pa) VALUES ('0000000014000000000000000200000002000000030000000500060000000200000003000000050001');
 
+SELECT Sum(PC_NumPoints(pa)) FROM pa_test_dim;
+SELECT Sum(PC_MemSize(pa)) FROM pa_test_dim;
+SELECT Sum(PC_MemSize(pt)) FROM pt_test;
+SELECT Sum(PC_PatchMax(pa,'x')) FROM pa_test_dim;
+SELECT Sum(PC_PatchMin(pa,'x')) FROM pa_test_dim;
+
 
 --DROP TABLE pts_collection;
 --DROP TABLE pt_test;

pgsql_postgis/pointcloud_postgis--1.0.sql

@@ -42,14 +42,14 @@ CREATE CAST (pcpoint AS geometry) WITH FUNCTION geometry(pcpoint);
 -- Function to overlap polygon on patch
 --
 CREATE OR REPLACE FUNCTION PC_Intersects(pcpatch, geometry)
-    RETURNS pcpatch AS
+    RETURNS boolean AS
     $$
         SELECT ST_Intersects($2, geometry($1))
     $$ 
     LANGUAGE 'sql';
 
 CREATE OR REPLACE FUNCTION PC_Intersects(geometry, pcpatch)
-    RETURNS pcpatch AS
+    RETURNS boolean AS
     $$
         SELECT PC_Intersects($2, $1)
     $$