程序员笔记: October 2015

Friday, October 30, 2015

tilestream

============================================================================

Therein, Tom MacWright of the Mapbox project suggested a work-around of basically clicked over a tile to determine what the correct URL is. It appears the server reformulates the URL, possibly depending upon how many mbtiles are in the directly. I don't think this URL change is documented, so the work-around is to look at the generic instances, e.g. : http://192.168.1.25:8888/map/John_test2 to see what the URL root path is for a sample PNG. In my case, I had two mbtiles in the directory, and the URL for a PNG was, for example: http://themis:8888/v2/John_test2/17/21010/50421.png. Consequently, there are two issues here: 1) one must use the addLayers mechanism to import maps not served by mapbox.com, 2) the URLs are programmatically changed and for now (I could not find any documentation on the matter), the work-around is to access the mbtiles file directly and determine what the URL construct is by right clicking an image and viewing its particular URL.

Here's my HTML that used to successfully render within a mapbox.map object:

var map = L.mapbox.map('map');
    //map.addLayer(L.tileLayer('http://{s}.tile.stamen.com/toner/{z}/{x}/{y}.png'));  // original
    //
    //map.addLayer(L.tileLayer('http://192.168.1.25:8888/map/John_test2/{z}/{x}/{y}.png'));   // fails

    // per suggestion
    // http://support.mapbox.com/discussions/mapboxjs/1875-deploying-sqlite-database-via-tilestream
    map.addLayer(L.tileLayer('http://themis:8888/v2/John_test2/{z}/{x}/{y}.png'));

    //map.setView([0, 0], 5);  //original
    map.setView([38.297619, -122.296316 ], 17);




    
    
    <!--[if lte IE 8]>
    
    <![endif]

-->
Note: I changed my server host name from an IP to "themis", hence the difference there is a result of my changing the initialization of the tilestream server.

========================================
start tilestream on windows. open cmd, into directory.

node index.js

C:\jh\tilestream>node index.js
Creating tiles dir C:\Users\hu_j\Documents\MapBox\tiles
Started [Server Tile].
Started [Server Core:8888].

=======================================================

0) install node js, must be 0.10.x or 0.8.x( 4.2.1 will fail the build)
Must read install doc, follow exactly what version to use.

1)before install tilestream, you must install python(2.7, 3.5 is not support by node-gyp)
currently node-gyp support python 2.5 - 3.0

2)add python to system path environment variable, system variable -> path, add c:\python27\
the user variable PATH only for current user, so should use system variable for all users.
Restart needed to have system path take effect.

3) uninstall any pieces of visual studio 2005 2008 redistribution x86 x64, uninstall other python version. only left 2.7.

4) open cmd, You can use npm to install TileStream:

git clone https://github.com/mapbox/tilestream.git
cd tilestream
npm install

5) try to run the command npm install , the error:
c:\tilestream>npm install

Error: ENOENT, stat 'C:\Users\hoogw\AppData\Roaming\npm'

Error: ENOENT, stat 'C:\Users\RT\AppData\Roaming\npm

Solution: Manually creating a folder named 'npm' in the displayed path fixed the problem.

`Error: ENOENT, stat 'C:\Users\\AppData\Roaming\npm'` on Windows 7

This is a consequence of joyent/node#8141, and is an issue with the Node installer for Windows. The workaround is to ensure that C:\Users\\AppData\Roaming\npm exists and is writable with your normal user account.

AppData is hidden folder, so

Open Folder Options by clicking the Start button , clicking Control Panel, clicking Appearance and Personalization, and then clicking Folder Options.
Click the View tab.
Under Advanced settings, click Show hidden files, folders, and drives, and then click OK.

6) Start TileStream: under tilestream folder

c:\tilestream>node index.js

7) error missing model
c:\tilestream\node_modules\connect\node_modules\qs\lib\index.js:5
const Stringify = require('./stringify');

window 7 and windows server 2012,
install tilestream without any error message.
But when I run it, it error out.
To verify this is bug, I did both windows 7 and server 2012, both give the exact same error.

c:\tilestream>node index.js

error:

c:\tilestream\node_modules\connect\node_modules\qs\lib\index.js:5
const Stringify = require('./stringify');
^^^^^
SyntaxError: Use of const in strict mode.
at Module._compile (module.js:439:25)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (c:\tilestream\node_modules\connect\lib\middleware\bod
yParser.js:13:10)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)

I found the problem is in connect folder, C:\tilestream\node_modules\connect, if I replace connect folder with the previously working copy's connect folder, it will works. If I use "npm install" then node index.js, will always get error message above

I just ran into this as well using node 0.10.36 (CentOS 6 EPEL packages). Using the --harmony option fixes it in my case:

$ node --harmony index.js

Done.

===================================================================
Must install vs c++, not the regular visual studio for web. otherwise will get "gyp build error"
Microsoft Visual Studio C++ 2012 Express version, visual studio 2012 express for desktop

- Python (v2.7.3 recommended, v3.x.x is not supported)
  - Make sure that you have a PYTHON environment variable, and it is set to drive:\path\to\python.exe not to a folder
- Windows XP/Vista/7:
  - Microsoft Visual Studio C++ 2013 (Express version works well)
    - If the install fails, try uninstalling any C++ 2010 x64&x86 Redistributable that you have installed first
    - If you get errors that the 64-bit compilers are not installed you may also need the compiler update for the Windows SDK 7.1
- Windows 7/8:
  - Microsoft Visual Studio C++ 2013 for Windows Desktop (Express version works well)
- All Windows Versions
  - For 64-bit builds of node and native modules you will also need the Windows 7 64-bit SDK
  - You may need to run one of the following commands if your build complains about WindowsSDKDir not being set, and you are sure you have already installed the SDK:


call "C:\Program Files\Microsoft SDKs\Windows\v7.1\bin\Setenv.cmd" /Release /x86

call "C:\Program Files\Microsoft SDKs\Windows\v7.1\bin\Setenv.cmd" /Release /x64

If you have multiple Python versions installed, you can identify which Python version node-gyp uses by setting the '--python' variable:


$ node-gyp --python /path/to/python2.7

If node-gyp is called by way of npm and you have multiple versions of Python installed, then you can set npm's 'python' config key to the appropriate value:


$ npm config set python /path/to/executable/python2.7

Note that OS X is just a flavour of Unix and so needs python, make, and C/C++. An easy way to obtain these is to install XCode from Apple, and then use it to install the command line tools (under Preferences -> Downloads).

4) specify "--msvs_version=2012"

  for example:   npm install --msvs_version=2012

5)

when install tilestream, run node.js command windows as adminstrator.

if error, look the error message, if you missing some model.

-------------------------------------------------
if the error is can't find python,
because missing node-gyp, you need to install it by:

Install node-gyp: npm install -g node-gyp

..........................................

Some npm plugins need node-gyp to be installed.

However, node-gyp has it's own dependencies (from the github page):

So to install it on windows you must first :

Install python: http://www.python.org/getit/windows/
Install Microsoft Visual Studio C++ 2012 Express version: http://go.microsoft.com/?linkid=9816758
Install node-gyp: npm install -g node-gyp

If you use linux (debian):

python / make are pre-installed!
just install gcc: sudo apt-get install g++
and then install node-gyp: sudo npm install -g node-gyp

---------------------------------------------------------------------

if the error is missing sqlite3, need install sqlite3

C:\jh\tilestream>npm install sqlite3

other model might missing

npm install sqlite3
npm install mbtiles
npm install jsdom
npm install uglify-js
npm install connect
npm install qs
npm install mime

Friday, October 23, 2015

15 Free Bootstrap Admin Themes

Free Bootstrap Admin Themes

Rendering big geodata on the fly with GeoJSON-VT

Despite the amazing advancements of computing technologies in recent years, processing and displaying large amounts of data dynamically is still a daunting, complex task. However, a smart approach with a good algorithmic foundation can enable things that were considered impossible before.

Let’s see if Mapbox GL JS can handle loading a 106 MB GeoJSON dataset of US ZIP code areas with 33,000+ features shaped by 5.4+ million points directly in the browser (without server support):

Wait, what?! A few seconds loading the data, and you can browse the whole data set smoothly and seamlessly. But how exactly does that work? Let’s find out.

Slicing vector tiles

It isn’t possible to render such a crazy amount of data in its entirety at 60 frames per second, but luckily, we don’t have to:

at lower zoom levels, shapes don’t need to be as detailed
at higher zoom levels, a lot of data is off-screen

The best way to optimize the data for all zoom levels and screens is to cut it into vector tiles. Traditionally, this is done on the server, using tools like Mapnik and PostGIS.

Could we create vector tiles on the fly, in the browser? Specifically for this purpose, I wrote a new JavaScript library — geojson-vt.

It turned out to be crazy fast, with its usefulness going way beyond the browser:

It powers GeoJSON rendering in Mapbox GL JS.
It was ported to C++14 to power on-the-fly point and polyline rendering on mobile devices with Mapbox Mobile.
It works on a server with Node.js, so we’re experimenting with using it in our server-side data-processing pipeline.

How GeoJSON-VT works

Preprocessing

Generating tiles involves many computationally expensive steps:

projecting geographic coordinates into screen coordinates
optimizing polyline and polygon detail for each zoom level
filtering out features that are too small for a particular zoom
clipping data to tile extents

GeoJSON-VT performs these steps in order: it projects the entire dataset and calculates simplification before creating tiles. This way, the creation of many tiles over many zoom levels can be run quickly and without duplication of work.

The first step of preprocessing is projecting all coordinates into [0..1, 0..1] range with the Web Mercator projection. Later, we can get the tile screen coordinates with a simple multiplication plus rounding:

screen_x = [tile_size × (2zoom x − tile_x)]
screen_y = [tile_size × (2zoom y − tile_y)]

The next step is calculating simplification data that is later used to reduce detail in each tile. It uses a modified Ramer-Douglas-Peucker simplification algorithm that recursively finds the most “significant” points in a polyline and discards unnecessary detail.

This is the same technique used in Leaflet and simplify-js, but instead of immediately simplifying data, GeoJSON-VT marks each vertex with an importance value, so it can be quickly included or excluded at a zoom level.

Because pixel distances scale linearly with 2zoom, we can run simplification once for the data and use the same values to optimize detail on every zoom level, filtering out unnecessary vertices almost instantly.

Then we calculate the area of each polygon ring and the length of each polyline. Areas and lengths are scaled with zoom level in the same way as distances, which allows us to filter out geometries that are too tiny to display on any particular zoom level using these pre-calculated values in linear time.

Finally, we save the bounding box of each feature, which will allow us to do clipping much faster.

Slicing the data into tiles

Slicing tiles involves clipping data to a square extent, but we can’t afford running a clipping algorithm on the whole dataset for every single tile — that would be too slow.

To make it fast, GeoJSON-VT uses a top-down, divide & conquer approach to clipping:

Slice the top z0 tile into four z1 tiles.
Slice each z1 tile into four z2 tiles.
Repeat recursively.

This makes slicing extremely fast, because each tile only needs to clip the features in its parent tile, rather than the entire dataset. But we can do even better than that.

Here’s how the most common rectangle clipping algorithm, Sutherland-Hodgeman, usually works:

To cut a tile out of a parent zoom tile, you need to cut the features by 4 half-planes formed by each of the axis-parallel red lines. For each line, we need to loop through all the points from the previous cut.

Note that the tile cut is not exactly a quarter of the parent tile, because each tile needs to have a small buffer that goes beyond what we display on a map to solve rendering issues (e.g. Mapbox Studio Classic buffers each 512px tile by 8 pixels in each direction by default).

Let’s roughly calculate how long it would take to clip out 4 tiles relative to the amount of data in the parent tile:

4 × (n + ½ n + ¼ n + ¼ n) = 8n

We can cut 4 tiles out of a parent tile in a smarter way. To do that, I made a simple modification of the half-plane clipping algorithm, making it clip shapes between two parallel-axis lines in one go: cutting out a “stripe”. This way, we can cut two halves with two vertical stripes first, then cut 2 tiles out of each half with two horizontal stripes:

Let’s see how this compares to the previous approach:

n + n + 4 × ½ n = 4n

So we now have roughly 2 times faster clipping of each tile.

Finally, we can use the feature bounding boxes to our advantage. Imagine cutting the sample ZIP code data above into tiles on low zoom levels — for each tile, most zip code shapes will be either trivial accepts — included completely without clipping — or trivial rejects — throw away. If we have bounding boxes pre-calculated, we can handle trivial cases like this without rebuilding all geometries point-by-point, speeding everything up considerably.

Slicing on the fly

To serve vector tiles on the fly without a perceptible delay, ideally, we would generate all the vector tiles up to max zoom into cache, and then just serve the tiles instantly when requested. This may be fine for small datasets, but for anything bigger, the tile slicing routine will eventually run out of memory. On zoom level 15 alone, there are potentially 415 = ~1 billion tiles — that’s just too much to handle, especially for the browser.

So we can’t slice tiles to the max zoom on initial data load, but we can slice them up to a point where subsequent on-demand slicing takes so little time that the loading delay is imperceptible. GeoJSON-VT tackles this in several ways:

Initially, the tiles are generated to certain zoom level, leaving everything else for on-demand slicing.
When slicing tiles, we keep track of the number of points combined in every tile. Initial slicing stops when a tile goes below a certain threshold (100,000 points by default). If a tile contains few geometries, then its children can be generated quickly as needed.
When drilling down to a particular tile, GeoJSON-VT only slices one tile into 4 children tiles per zoom through the zoom range, caching all sliced tiles. This way drilling down to neighboring tiles is very fast because a much smaller parent tile will be cached from the previous tile drill-down.

I found the initial slicing defaults (z5 and 100k points) to be the sweet spot for most sample datasets I tested the algorithm on — initial processing is relatively fast, while subsequent drill-down is pretty much instant.

We also need to minimize the amount of memory used in each step of the algorithm. This is achieved by several things:

The original geometry is only kept in the highest-zoom tiles and is thrown away once a tile has been cut into 4 tiles of the next zoom level, keeping only the simplified data. This way the memory needed to store all the tiling data doesn’t grow exponentially when slicing deeper.
All recursion is eliminated with iterative algorithms. Both simplification and tile slicing (recursive in nature) are implemented using a simple array queue processed in a loop. This avoids the situation where geometry that you no longer need gets piled up in call stacks: it’s garbage-collected properly.
Transforming projected (0..1, 0..1) coordinates into screen coordinates always happens on demand, because building new arrays takes a lot of memory if done for all tiles as soon as they are sliced.
Solid squares filled to the tile extent (tiles fully inside a polygon) are never sliced further, because all of its children are identical to the parent regardless of zoom level.

Debugging and performance profiling

To debug and optimize the library, I wrote a simple debug page where you can drag and drop a GeoJSON file onto a barebones map where it is processed by GeoJSON-VT:

The library API also has a debug option with 2 logging levels, doing various levels of performance timings and logging during the tiling when specified. The GIF above demonstrates level 1, with level 2 outputting debug info for each individual tile. This helped tackling all the bottlenecks and tracking issues tremendously.

Jank-free rendering in Mapbox GL JS

Despite the library being extremely fast, loading and processing big amounts of data still takes a significant amount of CPU time.

Browsers normally run JavaScript code in the same thread that handle all UI updates and user interaction. If JavaScript is busy doing heavy computation, the page will freeze and won’t accept any user input.

To avoid lags and freezes when loading and processing data, Mapbox GL JS utilizes Web Workers, a modern browser feature that permits running computationally-intensive code in the background without interrupting the main thread.

There’s one more problem though — transferring big amounts of data between a web worker and the main thread usually involves copying the data in memory, also causing the main thread to freeze.

To avoid that bottleneck, we take advantage of another HTML5 feature — Transferable Objects. It allows transferring data between threads instantly with zero copy if you pass the data in the form of typed arrays. This allows us to provide a smooth, jank-free experience despite plenty of heavy CPU work happening in the background.

Using the library

If you’re using Mapbox GL-based tools (either GL JS or Mapbox Mobile), you’re already using GeoJSON-VT under the hood.

You can use it manually for other applications too — all the complexity of the processing is hidden behind a very simple API:

var tileIndex = geojsonvt(data);
...
var tile = tileIndex.getTile(z, x, y);

The resulting tile is a simple JSON equivalent of Vector Tile, and can be rendered with ease.

Try it! We’re curious to see what other creative uses of the library you can come up with. I also encourage you to check out the geojson-vt repository and read through the code — it’s was carefully written to be as simple and clean as possible.

Thanks for bearing with me in this long technical article, and don’t hesitate to hit me up on Twitterif you have any questions or comments.

Friday, October 30, 2015

tilestream

Error: ENOENT, stat 'C:\Users\\AppData\Roaming\npm' on Windows 7

So to install it on windows you must first :

If you use linux (debian):

Friday, October 23, 2015

15 Free Bootstrap Admin Themes

Free Bootstrap Admin Themes

DevOOPS v.2 – A Modern Bootstrap Dashboard Theme

Blocks – A Single-Page Widgetised Admin Template

AdminLTE – A Clean Admin Theme

Gentelella Bootstrap Admin Template

Responsive Dashboard (AngularJS & Bootstrap)

Metro Dashboard – A Modern and Clean Admin Template

Metis – A Bootstrap Admin Template

SB Admin v.2 – A Minimal Bootstrap Dashboard Theme

TemplateVamp – Bootstrap Admin Theme

Clean Bootstrap Dashboard Theme

Charisma Responsive Admin Template

HTML5 Admin Template

Meritoo – A Minimal Admin Theme

Hierapolis – A Bold Bootstrap Admin Template

Bootstrap Admin Theme

EdenAdmin – A Minimal Boostrap Dashooard Theme