read_parquet() browser support #18

Activity

1. yharby commented on Mar 14, 2024

   

   yharby

   on Mar 14, 2024

   Last edited by yharby

   Owner

   More actions

   @jwass, thank you so much! Yes, the Content Security Policy in the browser, I believe, doesn't allow the wildcard/glob (https://*) functions to be used because the entire app is running on the client side in the browser, with no servers involved (ref: duckdb/duckdb-wasm#1040 ). You can try:

   SELECT 
       id,
       geometry
   FROM 
       's3://overturemaps-us-west-2/release/2024-03-12-alpha.0/theme=buildings/type=building/part-00150-4dfc75cd-2680-4d52-b5e0-f4cc9f36b267-c000.zstd.parquet'
   LIMIT 
       100;
   

   Or, I believe you can create a list with all the parquet files you want to query from, like:

   SELECT 
       id,
       geometry
   FROM 
       read_parquet(['s3://overturemaps-us-west-2/release/2024-03-12-alpha.0/theme=buildings/type=building/part-00150-4dfc75cd-2680-4d52-b5e0-f4cc9f36b267-c000.zstd.parquet','s3://overturemaps-us-west-2/release/2024-03-12-alpha.0/theme=buildings/type=building/part-00151-4dfc75cd-2680-4d52-b5e0-f4cc9f36b267-c000.zstd.parquet'])
   WHERE
       bbox.minx > 78.53608051444496 
   AND bbox.maxx < 78.56248512255797 
   AND bbox.miny > 25.483406547011967 
   AND bbox.maxy < 25.521748210549873
   LIMIT 
       2000;
   

   We can implement a workaround in this app by utilizing STAC for the GeoParquet files stored in S3. This way, when a user zooms into a specific bounding box (bbox), we can intersect this bbox with the ones defined in the STAC catalog. This process will enable us to list the GeoParquet files containing data for the specified region.
2. jwass commented on Mar 14, 2024

   

   jwass

   on Mar 14, 2024

   Author

   More actions

   Thanks! I literally just discovered this about 10 minutes ago and came to say that it works for individual files. This is pretty cool!

   ❤️React with ❤️1Reacted by Youssef Harby