Updates of Sentinel data

Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Updates of Sentinel data

Grega Milcinski
Dear all,

there have been several questions raised in the recent period about the changes in the Sentinel data. We decided to make use of this mailing list to elaborate a bit on what is happening. We would also suggest we (and you) use this mailing list more often so that information spread wider.

There have been several issues in the past, which affected distribution of Sentinel-2 data:

1) Single-tile packages
End of September Copernicus decided to distribute data in single-tile packages rather than “products compiled of about 10 tiles”. When building Amazon archive we decided at the beginning to do it in this way due to better organisation of data and optimisation of download traffic.  The change did therefore not affect Amazon archive at all and data are stored consistently with the archive data. 
The fact that “product metadata” is now directly aligned with “tile metadata”, there is a bit of redundancy in Amazon archive’s structure - e.g. we keep product and tile meta-data separated (to avoid duplication of data in multi-tile products). However, we decided not to change this in order to keep the structure consistent.
It seems that this change resulted in additional stress to SciHub/ApiHub platforms, which resulted in additional delays in the distribution. We worked around connection problems and sync seems to be stable. However, we still notice, that there are significant delays in ingestion of data, as described here:

You can observe this in the attached pictures showing number of products distributed with delay less than 24 hours and more than 24 hours.

2) Change of the tile structure
End of October/beginning of November Copernicus will change the filenames to avoid too-long names. This is also something we have planned from the early beginning and looking at sample data it seems that new filenames will be similar to what we have at AWS. We therefore do not expect significant effect on data and we are almost certain that we will be able to maintain a consistent file structure. 
The change itself will obviously affect our ingestion procedures, which will result in a short delay of SciHub-AWS sync - e.g. for a day or two, depending on the actual mess that is caused by these changes.

The possible positive effect might be that SNAP will work with the AWS structure. We are not sure about that but since change is moving towards AWS direction, it might be the case.

In addition to change of names there is a new product to be included - “True color image”, which might be a good candidate to show as true image. Based on the sample image it seems that it is not much else than RGB bands multiplied by a factor of 3, but perhaps there will be better color correction in place.

Regards
Grega




--------------------------------------------------------------------
Grega Milcinski                  Sinergise
CEO                              http://www.sinergise.com
                                              Cvetkova ulica 29
Email: [hidden email]   SI-1000 Ljubljana
Phone: +386 (40) 427-642                Slovenia
--------------------------------------------------------------------


_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds
Reply | Threaded
Open this post in threaded view
|

Re: Updates of Sentinel data

Flasher, Joe
Thanks for this update Grega and I’d like to echo your call to use this list for more information distribution. 

I also wanted to let the group know that we will be removing the top-level `zips` folder in the `sentinel-s2-l1c` S3 bucket. We will be instead providing these on a 7-day rolling basis in the `sentinel-s2-l1c-zips` bucket which will be marked as a requester pays bucket (http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html).

On Oct 14, 2016, at 4:31 PM, Grega Milcinski <[hidden email]> wrote:

Dear all,

there have been several questions raised in the recent period about the changes in the Sentinel data. We decided to make use of this mailing list to elaborate a bit on what is happening. We would also suggest we (and you) use this mailing list more often so that information spread wider.

There have been several issues in the past, which affected distribution of Sentinel-2 data:

1) Single-tile packages
End of September Copernicus decided to distribute data in single-tile packages rather than “products compiled of about 10 tiles”. When building Amazon archive we decided at the beginning to do it in this way due to better organisation of data and optimisation of download traffic.  The change did therefore not affect Amazon archive at all and data are stored consistently with the archive data. 
The fact that “product metadata” is now directly aligned with “tile metadata”, there is a bit of redundancy in Amazon archive’s structure - e.g. we keep product and tile meta-data separated (to avoid duplication of data in multi-tile products). However, we decided not to change this in order to keep the structure consistent.
It seems that this change resulted in additional stress to SciHub/ApiHub platforms, which resulted in additional delays in the distribution. We worked around connection problems and sync seems to be stable. However, we still notice, that there are significant delays in ingestion of data, as described here:

You can observe this in the attached pictures showing number of products distributed with delay less than 24 hours and more than 24 hours.

2) Change of the tile structure
End of October/beginning of November Copernicus will change the filenames to avoid too-long names. This is also something we have planned from the early beginning and looking at sample data it seems that new filenames will be similar to what we have at AWS. We therefore do not expect significant effect on data and we are almost certain that we will be able to maintain a consistent file structure. 
The change itself will obviously affect our ingestion procedures, which will result in a short delay of SciHub-AWS sync - e.g. for a day or two, depending on the actual mess that is caused by these changes.

The possible positive effect might be that SNAP will work with the AWS structure. We are not sure about that but since change is moving towards AWS direction, it might be the case.

In addition to change of names there is a new product to be included - “True color image”, which might be a good candidate to show as true image. Based on the sample image it seems that it is not much else than RGB bands multiplied by a factor of 3, but perhaps there will be better color correction in place.

Regards
Grega
<more_than_24h.png><less_than_24h.png>




--------------------------------------------------------------------
Grega Milcinski                  Sinergise
CEO                              http://www.sinergise.com
                                              Cvetkova ulica 29
Email: [hidden email]   SI-1000 Ljubljana
Phone: +386 (40) 427-642                Slovenia
--------------------------------------------------------------------

_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds


_______________________________________________
Sentinel-pds mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/sentinel-pds
Reply | Threaded
Open this post in threaded view
|

Change of access permissions to S-2 L1C bucket

Grega Milcinski
Dear Sentinel-2 AWS Public Datasets user,

by end of June 2018 we will set Sentinel-2 L1C dataset as “Requester pays" in a manner similar to Sentinel-2 L2A and Sentinel-1 GRD were from the beginning. This may affect your workflow.
Visit this page:
https://forum.sentinel-hub.com/t/changes-of-the-access-rights-to-l1c-bucket-at-aws-public-datasets-requester-pays/172
to find out details and how to embrace the changes.

We would also welcome your feedback on the forum to make this change as seamless as possible for your applications

Best,
Grega
_______________________________________________
Sentinel-pds mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sentinel-pds