Quantcast

Sentinel 2 CSV Extract

Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Sentinel 2 CSV Extract

Chris Brown-2
Hi All,

I was wondering if it is possible to publish a CSV extract of Sentinel 2 scenes/images with the publicly hosted imagery on S3 similar to the extract for Landsat 8.

Azavea has built an ingest pipeline for indexing the Sentinel 2 data for an image/satellite processing service. In order to determine if there is new imagery to ingest (and verify that we have ingested everything) we exploit the directory structure to get the relevant JSON metadata files for each day (e.g. 2016/12/11/<product>/productInfo.json), get the list of tiles, and then go from there.

A CSV extract would help us verify that we are not missing files and determine what needs to be indexed for our purposes more efficiently.

Best,

Chris

--
Chris Brown, GIS Software Developer

Azavea |  990 Spring Garden Street, 5th Floor, Philadelphia, PA

[hidden email] | P 215.558.6191 | F 215.925.2663
Web azavea.com  |  Blog azavea.com/blogs  | Twitter @azavea

_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sentinel 2 CSV Extract

Grega Milcinski
Hi Chris,

there is no CSV export available currently, sorry about that.
If you have resources available, it would be welcome if you could implement this part and push it to the Sentinel Ingestor:
https://github.com/sinergise/Sentinel2ProductIngestor/
We will then be happy to deploy it.

However, to determine if there is a new imagery to ingest, I would recommend to subscribe to AWS notifications, see “Notification service for new products” section in the page above. In this way you will get an alert immediately after the image is ingested.

Regards
Grega


> Hi All,
>
> I was wondering if it is possible to publish a CSV extract of Sentinel 2 scenes/images with the publicly hosted imagery on S3 similar to the extract for Landsat 8.
>
> Azavea has built an ingest pipeline for indexing the Sentinel 2 data for an image/satellite processing service. In order to determine if there is new imagery to ingest (and verify that we have ingested everything) we exploit the directory structure to get the relevant JSON metadata files for each day (e.g. 2016/12/11/<product>/productInfo.json), get the list of tiles, and then go from there.
>
> A CSV extract would help us verify that we are not missing files and determine what needs to be indexed for our purposes more efficiently.
>
> Best,
>
> Chris
>
> --
> Chris Brown, GIS Software Developer
>
> Azavea |  990 Spring Garden Street, 5th Floor, Philadelphia, PA
> [hidden email] | P 215.558.6191 | F 215.925.2663
> Web azavea.com  |  Blog azavea.com/blogs  | Twitter @azavea
> _______________________________________________
> Sentinel-pds mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/sentinel-pds

_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sentinel 2 CSV Extract

Chris Brown-2
Thanks Grega.

I did not know about the SNS notification topic -- the Landsat 8 topic is a little more prominently displayed on the Earth on AWS page for Landsat.

I have not done much Java programming so I might not get to contributing to the project for a bit. In the meantime, I might just hook up a lambda function to the SNS topic and come up with some way to track the state/history.

Chris

On Tue, Dec 13, 2016 at 4:45 PM, Grega Milcinski <[hidden email]> wrote:
Hi Chris,

there is no CSV export available currently, sorry about that.
If you have resources available, it would be welcome if you could implement this part and push it to the Sentinel Ingestor:
https://github.com/sinergise/Sentinel2ProductIngestor/
We will then be happy to deploy it.

However, to determine if there is a new imagery to ingest, I would recommend to subscribe to AWS notifications, see “Notification service for new products” section in the page above. In this way you will get an alert immediately after the image is ingested.

Regards
Grega


> Hi All,
>
> I was wondering if it is possible to publish a CSV extract of Sentinel 2 scenes/images with the publicly hosted imagery on S3 similar to the extract for Landsat 8.
>
> Azavea has built an ingest pipeline for indexing the Sentinel 2 data for an image/satellite processing service. In order to determine if there is new imagery to ingest (and verify that we have ingested everything) we exploit the directory structure to get the relevant JSON metadata files for each day (e.g. 2016/12/11/<product>/productInfo.json), get the list of tiles, and then go from there.
>
> A CSV extract would help us verify that we are not missing files and determine what needs to be indexed for our purposes more efficiently.
>
> Best,
>
> Chris
>
> --
> Chris Brown, GIS Software Developer
>
> Azavea |  990 Spring Garden Street, 5th Floor, Philadelphia, PA
> [hidden email] | P <a href="tel:215.558.6191" value="+12155586191" target="_blank">215.558.6191 | F <a href="tel:215.925.2663" value="+12159252663" target="_blank">215.925.2663
> Web azavea.com  |  Blog azavea.com/blogs  | Twitter @azavea
> _______________________________________________
> Sentinel-pds mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/sentinel-pds

_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds



--
Chris Brown, GIS Software Developer

Azavea |  990 Spring Garden Street, 5th Floor, Philadelphia, PA

[hidden email] | P <a href="tel:(215)%20558-6191" value="+12155586191" target="_blank">215.558.6191 | F <a href="tel:(215)%20925-2663" value="+12159252663" target="_blank">215.925.2663
Web azavea.com  |  Blog azavea.com/blogs  | Twitter @azavea

_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sentinel 2 CSV Extract

Flasher, Joe
I can make the SNS topic more prominent on the AWS landing page. 

On Dec 14, 2016, at 07:29, Chris Brown <[hidden email]> wrote:

Thanks Grega.

I did not know about the SNS notification topic -- the Landsat 8 topic is a little more prominently displayed on the Earth on AWS page for Landsat.

I have not done much Java programming so I might not get to contributing to the project for a bit. In the meantime, I might just hook up a lambda function to the SNS topic and come up with some way to track the state/history.

Chris

On Tue, Dec 13, 2016 at 4:45 PM, Grega Milcinski <[hidden email]> wrote:
Hi Chris,

there is no CSV export available currently, sorry about that.
If you have resources available, it would be welcome if you could implement this part and push it to the Sentinel Ingestor:
https://github.com/sinergise/Sentinel2ProductIngestor/
We will then be happy to deploy it.

However, to determine if there is a new imagery to ingest, I would recommend to subscribe to AWS notifications, see “Notification service for new products” section in the page above. In this way you will get an alert immediately after the image is ingested.

Regards
Grega


> Hi All,
>
> I was wondering if it is possible to publish a CSV extract of Sentinel 2 scenes/images with the publicly hosted imagery on S3 similar to the extract for Landsat 8.
>
> Azavea has built an ingest pipeline for indexing the Sentinel 2 data for an image/satellite processing service. In order to determine if there is new imagery to ingest (and verify that we have ingested everything) we exploit the directory structure to get the relevant JSON metadata files for each day (e.g. 2016/12/11/<product>/productInfo.json), get the list of tiles, and then go from there.
>
> A CSV extract would help us verify that we are not missing files and determine what needs to be indexed for our purposes more efficiently.
>
> Best,
>
> Chris
>
> --
> Chris Brown, GIS Software Developer
>
> Azavea |  990 Spring Garden Street, 5th Floor, Philadelphia, PA
> [hidden email] | P <a href="tel:215.558.6191" value="&#43;12155586191" target="_blank"> 215.558.6191 | F <a href="tel:215.925.2663" value="&#43;12159252663" target="_blank"> 215.925.2663
> Web azavea.com  |  Blog azavea.com/blogs  | Twitter @azavea
> _______________________________________________
> Sentinel-pds mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/sentinel-pds

_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds



--
Chris Brown, GIS Software Developer

Azavea |  990 Spring Garden Street, 5th Floor, Philadelphia, PA

[hidden email] | P <a href="tel:(215)%20558-6191" value="&#43;12155586191" target="_blank">215.558.6191 | F <a href="tel:(215)%20925-2663" value="&#43;12159252663" target="_blank">215.925.2663
Web azavea.com  |  Blog azavea.com/blogs  | Twitter @azavea
_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds

_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds
Loading...