Wednesday, December 2, 2015

shape file shp geojson import mongodb




1)  install Chocolatey NuGet https://chocolatey.org/

     open cmd as admin, cd c:\

     copy and past after c:>

@powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin




2) install jq

      close and re-open cmd as admin

go to chocolatey folder by: 

cd C:\ProgramData\chocolatey

run

choco install jq --force -version 1.5


3)  go to where has geojson file

open cmd

cd C:\jh\other_map_geojson\Raw_geojson

run

jq --compact-output ".features[]" parcels.geojson > parcels0.geojson

parcels.geojson is input,  parcels0.geojson is output.




=========================================================

jq command CAN NOT do batch script, it error, not sure why, have to do one command by one command.

Some time jq error out, you have to do manually replace 


1) remove head and tail

2) remove last comma at each line.  by replace ]]]}, to ]]]},  point, line, polygon situation is different.
















==========================================================
import geojson to mongodb


Download jq (it's sed-like program but for JSON)
Then run:


jq --compact-output ".features[]" input.geojson > output.geojson


then


mongoimport --db dbname -c collectionname --file "output.geojson" --jsonArray






--jsonArray
Accepts the import of data expressed with multiple MongoDB documents within a single JSON array. Limited to imports of 16 MB or smaller.

If the file size larger than 16MB, you could do this
jq --compact-output ".features[]" input.geojson > output.geojson
This will give you exactly one line for one object, no comma at end.
{.....}
{.......}
{...}

{"type":"Feature","geometry":{"type":"Point","coordinates":[-80.87088507656375,35.21515162500578]},"properties":{"name":"ABBOTT NEIGHBORHOOD PARK","address":"1300  SPRUCE ST"}}
{"type":"Feature","geometry":{"type":"Point","coordinates":[-80.83775386582222,35.24980190252168]},"properties":{"name":"DOUBLE OAKS CENTER","address":"1326 WOODWARD AV"}}
{"type":"Feature","geometry":{"type":"Point","coordinates":[-80.83827000459532,35.25674709224663]},"properties":{"name":"DOUBLE OAKS NEIGHBORHOOD PARK","address":"2605  DOUBLE OAKS RD"}}
{"type":"Feature","geometry":{"type":"Point","coordinates":[-80.83697759172735,35.25751734669229]},"properties":{"name":"DOUBLE OAKS POOL","address":"1200 NEWLAND RD"}}
{"type":"Feature","geometry":{"type":"Point","coordinates":[-80.81647652154736,35.40148708491418]},"properties":{"name":"DAVID B. WAYMER FLYING REGIONAL PARK","address":"15401 HOLBROOKS RD"}}
{"type":"Feature","geometry":{"type":"Point","coordinates":[-80.83556459443902,35.39917224760999]},"properties":{"name":"DAVID B. WAYMER COMMUNITY PARK","address":"302 HOLBROOKS RD"}}
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[-80.72487831115721,35.26545403190955],[-80.72135925292969,35.26727607954368],[-80.71517944335938,35.26769654625573],[-80.7125186920166,35.27035945142482],[-80.70857048034668,35.268257165144064],[-80.70479393005371,35.268397319259996],[-80.70324897766113,35.26503355355979],[-80.71088790893555,35.2553619492954],[-80.71681022644043,35.2553619492954],[-80.7150936126709,35.26054831539319],[-80.71869850158691,35.26026797976481],[-80.72032928466797,35.26061839914875],[-80.72264671325684,35.26033806376283],[-80.72487831115721,35.26545403190955]]]},"properties":{"name":"Plaza Road Park"}}
mongoimport --db dbname -c collectionname --file "output.geojson" --jsonArray










----------------------------------------------------------------------------------------------------------------
this worked:
mongoimport --db mean-dev -c points --file "points.geojson" --jsonArray

Let me save you the hours lost in trying to figure out getting GeoJSON features into individual documents in a collection versus one document with nested features.
MongoDB 3.0.2. Valid GeoJSON file full of 809 features.
Using this code:
mongoimport --db test --type json --file ./'mobilemaps.geojson' 
I was able to create one document in the collection named 'mobilemaps'. That single document had all the features (points and descriptive data) correclty imported.
But, I wanted a collection of documents. I'm new to MongoDB, so I'm studying and thought that might work better for my needs. Could not figure out how to get a successful import. I had read about pulling out the surrounding object and array tags:
{
  "type": "FeatureCollection",
    "features": [
...
   ]
}
The cleaned file then generated errors upon import due to the comma between features. 
So, I created a file with my feature collection, stripped off the surrounding tags. 
Then I did a search and replace:
}
 },
{
replace (e.g., remove the comma and add a new line)
}
 }
{
Running mongoimport on the resulting file gave me 809 documents in the collection. 
=====================================================================

Right now you have an array of features. MongoDB will consider this to be one document. Try deleting the following from the beginning of your geojson:
{
"type": "FeatureCollection",
"features": [
Also, delete the following from the end of your geojson:
]
}
EDIT - Also, mongo expects one document per line. So make sure that your only \n is between documents! e.g.
...    
},\n
    {
        "type": "Feature",
        "id": "node/2664472516",
...

No comments: