Processing with raster files > 4GB

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Processing with raster files > 4GB

Andreas Neumann-4
Hi,

I am having a lot of troubles with large raster files and processing.

The intermediate results can go >4GB. Processing stops with not so
helpful error messages.

I wonder if processing is hitting the 4GB limit. Can I force processing
to use the BIGTIFF option or a raster format that can handle >4GB files?

Thank you,
Andreas
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Giovanni Manghi-2
> Hi,
>
> I am having a lot of troubles with large raster files and processing.
>
> The intermediate results can go >4GB. Processing stops with not so
> helpful error messages.
>
> I wonder if processing is hitting the 4GB limit. Can I force processing
> to use the BIGTIFF option or a raster format that can handle >4GB files?
>



Hi Andreas, what tool specifically?

cheers

-- Giovanni --
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Andreas Neumann-4
Hi,

I tried various tools, e.g.

* SAGA reclassify
* GRASS r.mapcalc

Both work fine on smaller data sets, but fail on big data sets.

This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM and
40GB disk free on C:, more free space on other disks where the data
resides.

I am not totally sure if it is the Bigtiff issue or some other issue. It
is just a guess that maybe I am hitting a limit with tiff. It is not
fully transparent to me what Processing is doing in the background (what
file formats with what options it creates).

It may also be an issue with SAGA or GRASS directly.

Thanks if you know anything,
Andreas

Am 2014-11-18 21:15, schrieb Giovanni Manghi:

>> Hi,
>>
>> I am having a lot of troubles with large raster files and processing.
>>
>> The intermediate results can go >4GB. Processing stops with not so
>> helpful error messages.
>>
>> I wonder if processing is hitting the 4GB limit. Can I force
>> processing
>> to use the BIGTIFF option or a raster format that can handle >4GB
>> files?
>>
>
>
>
> Hi Andreas, what tool specifically?
>
> cheers
>
> -- Giovanni --
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Carlos López PSIG
Hi Andreas,
I also had problems to reclassify BIGTIFF with SAGA and GRASS!
Finally I used GDAL!

Best regards,



Carlos López Quintanilla

www.psig.es
[hidden email]
+34 699.680.261




2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:
Hi,

I tried various tools, e.g.

* SAGA reclassify
* GRASS r.mapcalc

Both work fine on smaller data sets, but fail on big data sets.

This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM and 40GB disk free on C:, more free space on other disks where the data resides.

I am not totally sure if it is the Bigtiff issue or some other issue. It is just a guess that maybe I am hitting a limit with tiff. It is not fully transparent to me what Processing is doing in the background (what file formats with what options it creates).

It may also be an issue with SAGA or GRASS directly.

Thanks if you know anything,
Andreas

Am 2014-11-18 21:15, schrieb Giovanni Manghi:

Hi,

I am having a lot of troubles with large raster files and processing.

The intermediate results can go >4GB. Processing stops with not so
helpful error messages.

I wonder if processing is hitting the 4GB limit. Can I force processing
to use the BIGTIFF option or a raster format that can handle >4GB files?




Hi Andreas, what tool specifically?

cheers

-- Giovanni --
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer


_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Andreas Neumann-4
Hi Carlos,

GDAL would be my first choice - because it easy to automate. However, I
don't know how can do my task with GDAL.

 From my grayscale tiff images I want to replace one pixel value (249)
with white (255). All other pixels should stay at the value they have.

I came across https://github.com/chiatt/gdal_reclassify - but this
reclassifies all pixels, not only selected ones.

Maybe I could use http://www.gdal.org/gdal_calc.html with a conditional
syntax?

Thanks,
Andreas

Am 2014-11-19 09:10, schrieb Carlos López PSIG:

> Hi Andreas,
>
> I also had problems to reclassify BIGTIFF with SAGA and GRASS!
> Finally I used GDAL!
>
> Best regards,
>
>  [2]
>
> CARLOS LÓPEZ QUINTANILLA
>
> www.psig.es [2]
> [hidden email]
> +34 699.680.261
>
> 2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:
>
>> Hi,
>>
>> I tried various tools, e.g.
>>
>> * SAGA reclassify
>> * GRASS r.mapcalc
>>
>> Both work fine on smaller data sets, but fail on big data sets.
>>
>> This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM
>> and 40GB disk free on C:, more free space on other disks where the
>> data resides.
>>
>> I am not totally sure if it is the Bigtiff issue or some other
>> issue. It is just a guess that maybe I am hitting a limit with tiff.
>> It is not fully transparent to me what Processing is doing in the
>> background (what file formats with what options it creates).
>>
>> It may also be an issue with SAGA or GRASS directly.
>>
>> Thanks if you know anything,
>> Andreas
>>
>> Am 2014-11-18 21:15, schrieb Giovanni Manghi:
>>
>> Hi,
>>
>> I am having a lot of troubles with large raster files and
>> processing.
>>
>> The intermediate results can go >4GB. Processing stops with not so
>> helpful error messages.
>>
>> I wonder if processing is hitting the 4GB limit. Can I force
>> processing
>> to use the BIGTIFF option or a raster format that can handle >4GB
>> files?
>>
>> Hi Andreas, what tool specifically?
>>
>> cheers
>>
>> -- Giovanni --
>> _______________________________________________
>> Qgis-developer mailing list
>> [hidden email]
>> http://lists.osgeo.org/mailman/listinfo/qgis-developer [1]
>  _______________________________________________
>  Qgis-developer mailing list
>  [hidden email]
>  http://lists.osgeo.org/mailman/listinfo/qgis-developer [1]
>
>
>
> Links:
> ------
> [1] http://lists.osgeo.org/mailman/listinfo/qgis-developer
> [2] http://www.psig.es
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

giohappy


> From my grayscale tiff images I want to replace one pixel value (249) with white (255). All other pixels should stay at the value they have.
>
> I came across https://github.com/chiatt/gdal_reclassify - but this reclassifies all pixels, not only selected ones.
>
> Maybe I could use http://www.gdal.org/gdal_calc.html with a conditional syntax?

Hi Andreas, do you need to change the value of one specific pixel only? I mean the pixel at [x,y] index?

giovanni

>
> Thanks,
> Andreas
>
> Am 2014-11-19 09:10, schrieb Carlos López PSIG:
>>
>> Hi Andreas,
>>
>> I also had problems to reclassify BIGTIFF with SAGA and GRASS!
>> Finally I used GDAL!
>>
>> Best regards,
>>
>>  [2]
>>
>> CARLOS LÓPEZ QUINTANILLA
>>
>> www.psig.es [2]
>>
>> [hidden email]
>> +34 699.680.261
>>
>> 2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:
>>
>>> Hi,
>>>
>>> I tried various tools, e.g.
>>>
>>> * SAGA reclassify
>>> * GRASS r.mapcalc
>>>
>>> Both work fine on smaller data sets, but fail on big data sets.
>>>
>>> This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM
>>> and 40GB disk free on C:, more free space on other disks where the
>>> data resides.
>>>
>>> I am not totally sure if it is the Bigtiff issue or some other
>>> issue. It is just a guess that maybe I am hitting a limit with tiff.
>>> It is not fully transparent to me what Processing is doing in the
>>> background (what file formats with what options it creates).
>>>
>>> It may also be an issue with SAGA or GRASS directly.
>>>
>>> Thanks if you know anything,
>>> Andreas
>>>
>>> Am 2014-11-18 21:15, schrieb Giovanni Manghi:
>>>
>>> Hi,
>>>
>>> I am having a lot of troubles with large raster files and
>>> processing.
>>>
>>> The intermediate results can go >4GB. Processing stops with not so
>>> helpful error messages.
>>>
>>> I wonder if processing is hitting the 4GB limit. Can I force
>>> processing
>>> to use the BIGTIFF option or a raster format that can handle >4GB
>>> files?
>>>
>>> Hi Andreas, what tool specifically?
>>>
>>> cheers
>>>
>>> -- Giovanni --
>>> _______________________________________________
>>> Qgis-developer mailing list
>>> [hidden email]
>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer [1]
>>
>>  _______________________________________________
>>  Qgis-developer mailing list
>>  [hidden email]
>>  http://lists.osgeo.org/mailman/listinfo/qgis-developer [1]
>>
>>
>>
>> Links:
>> ------
>> [1] http://lists.osgeo.org/mailman/listinfo/qgis-developer
>> [2] http://www.psig.es
>
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer


_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

roy roy
In reply to this post by Andreas Neumann-4
Hi Andreas,

you can surely try to import the raster directly into GRASS in a 64bit
linux,
very often this overcome memory issues, and use mapcalc sintiax,
something like:

reclassed_raster = if(original_raster == 249,255,original_raster)

regards,

Roy.

>
> GDAL would be my first choice - because it easy to automate. However,
> I don't know how can do my task with GDAL.
>
> From my grayscale tiff images I want to replace one pixel value (249)
> with white (255). All other pixels should stay at the value they have.
>
> I came across https://github.com/chiatt/gdal_reclassify - but this
> reclassifies all pixels, not only selected ones.
>
> Maybe I could use http://www.gdal.org/gdal_calc.html with a
> conditional syntax?
>
> Thanks,
> Andreas
>
> Am 2014-11-19 09:10, schrieb Carlos López PSIG:
>> Hi Andreas,
>>
>> I also had problems to reclassify BIGTIFF with SAGA and GRASS!
>> Finally I used GDAL!
>>
>> Best regards,
>>
>>  [2]
>>
>> CARLOS LÓPEZ QUINTANILLA
>>
>> www.psig.es [2]
>> [hidden email]
>> +34 699.680.261
>>
>> 2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:
>>
>>> Hi,
>>>
>>> I tried various tools, e.g.
>>>
>>> * SAGA reclassify
>>> * GRASS r.mapcalc
>>>
>>> Both work fine on smaller data sets, but fail on big data sets.
>>>
>>> This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM
>>> and 40GB disk free on C:, more free space on other disks where the
>>> data resides.
>>>
>>> I am not totally sure if it is the Bigtiff issue or some other
>>> issue. It is just a guess that maybe I am hitting a limit with tiff.
>>> It is not fully transparent to me what Processing is doing in the
>>> background (what file formats with what options it creates).
>>>
>>> It may also be an issue with SAGA or GRASS directly.
>>>
>>> Thanks if you know anything,
>>> Andreas
>>>
>>> Am 2014-11-18 21:15, schrieb Giovanni Manghi:
>>>
>>> Hi,
>>>
>>> I am having a lot of troubles with large raster files and
>>> processing.
>>>
>>> The intermediate results can go >4GB. Processing stops with not so
>>> helpful error messages.
>>>
>>> I wonder if processing is hitting the 4GB limit. Can I force
>>> processing
>>> to use the BIGTIFF option or a raster format that can handle >4GB
>>> files?
>>>
>>> Hi Andreas, what tool specifically?
>>>
>>> cheers
>>>
>>> -- Giovanni --
>>> _______________________________________________
>>> Qgis-developer mailing list
>>> [hidden email]
>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer [1]
>>  _______________________________________________
>>  Qgis-developer mailing list
>>  [hidden email]
>>  http://lists.osgeo.org/mailman/listinfo/qgis-developer [1]
>>
>>
>>
>> Links:
>> ------
>> [1] http://lists.osgeo.org/mailman/listinfo/qgis-developer
>> [2] http://www.psig.es
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Andreas Neumann-4
In reply to this post by giohappy
Hi Giovanni,

I need to change all pixels that have the value of 249 to a value of
255. All other pixels should stay at their value.

Andreas

Am 2014-11-19 10:21, schrieb G. Allegri:

>> From my grayscale tiff images I want to replace one pixel value
> (249) with white (255). All other pixels should stay at the value they
> have.
>  >
>  > I came across https://github.com/chiatt/gdal_reclassify [1] - but
> this reclassifies all pixels, not only selected ones.
>  >
>  > Maybe I could use http://www.gdal.org/gdal_calc.html [2] with a
> conditional syntax?
>
> Hi Andreas, do you need to change the value of one specific pixel
> only? I mean the pixel at [x,y] index?
>
> giovanni
>
>>
>  > Thanks,
>  > Andreas
>  >
>  > Am 2014-11-19 09:10, schrieb Carlos López PSIG:
>  >>
>  >> Hi Andreas,
>  >>
>  >> I also had problems to reclassify BIGTIFF with SAGA and GRASS!
>  >> Finally I used GDAL!
>  >>
>  >> Best regards,
>  >>
>  >>  [2]
>  >>
>  >> CARLOS LÓPEZ QUINTANILLA
>  >>
>  >> www.psig.es [3] [2]
>  >>
>  >> [hidden email]
>  >> +34 699.680.261
>  >>
>  >> 2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:
>  >>
>  >>> Hi,
>  >>>
>  >>> I tried various tools, e.g.
>  >>>
>  >>> * SAGA reclassify
>  >>> * GRASS r.mapcalc
>  >>>
>  >>> Both work fine on smaller data sets, but fail on big data sets.
>  >>>
>  >>> This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM
>  >>> and 40GB disk free on C:, more free space on other disks where
> the
>  >>> data resides.
>  >>>
>  >>> I am not totally sure if it is the Bigtiff issue or some other
>  >>> issue. It is just a guess that maybe I am hitting a limit with
> tiff.
>  >>> It is not fully transparent to me what Processing is doing in the
>  >>> background (what file formats with what options it creates).
>  >>>
>  >>> It may also be an issue with SAGA or GRASS directly.
>  >>>
>  >>> Thanks if you know anything,
>  >>> Andreas
>  >>>
>  >>> Am 2014-11-18 21:15, schrieb Giovanni Manghi:
>  >>>
>  >>> Hi,
>  >>>
>  >>> I am having a lot of troubles with large raster files and
>  >>> processing.
>  >>>
>  >>> The intermediate results can go >4GB. Processing stops with not
> so
>  >>> helpful error messages.
>  >>>
>  >>> I wonder if processing is hitting the 4GB limit. Can I force
>  >>> processing
>  >>> to use the BIGTIFF option or a raster format that can handle >4GB
>  >>> files?
>  >>>
>  >>> Hi Andreas, what tool specifically?
>  >>>
>  >>> cheers
>  >>>
>  >>> -- Giovanni --
>  >>> _______________________________________________
>  >>> Qgis-developer mailing list
>  >>> [hidden email]
>  >>> http://lists.osgeo.org/mailman/listinfo/qgis-developer [4] [1]
>  >>
>  >>  _______________________________________________
>  >>  Qgis-developer mailing list
>  >>  [hidden email]
>  >>  http://lists.osgeo.org/mailman/listinfo/qgis-developer [4] [1]
>  >>
>  >>
>  >>
>  >> Links:
>  >> ------
>  >> [1] http://lists.osgeo.org/mailman/listinfo/qgis-developer [4]
>  >> [2] http://www.psig.es [3]
>  >
>  > _______________________________________________
>  > Qgis-developer mailing list
>  > [hidden email]
>  > http://lists.osgeo.org/mailman/listinfo/qgis-developer [4]
>
> Links:
> ------
> [1] https://github.com/chiatt/gdal_reclassify
> [2] http://www.gdal.org/gdal_calc.html
> [3] http://www.psig.es
> [4] http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Carlos López PSIG
In reply to this post by Andreas Neumann-4
Hi Andreas,

Yes, you can use gdal_calc.py
Certainly you need to perform several processes, but are very fast!

Be careful with bands!
Perhaps you will want to separate them.

Ciao,




Carlos López Quintanilla

www.psig.es
[hidden email]
+34 699.680.261




2014-11-19 9:26 GMT+01:00 Andreas Neumann <[hidden email]>:
Hi Carlos,

GDAL would be my first choice - because it easy to automate. However, I don't know how can do my task with GDAL.

From my grayscale tiff images I want to replace one pixel value (249) with white (255). All other pixels should stay at the value they have.

I came across https://github.com/chiatt/gdal_reclassify - but this reclassifies all pixels, not only selected ones.

Maybe I could use http://www.gdal.org/gdal_calc.html with a conditional syntax?

Thanks,
Andreas

Am 2014-11-19 09:10, schrieb Carlos López PSIG:
Hi Andreas,

I also had problems to reclassify BIGTIFF with SAGA and GRASS!
Finally I used GDAL!

Best regards,

 [2]

CARLOS LÓPEZ QUINTANILLA

www.psig.es [2]

[hidden email]
<a href="tel:%2B34%20699.680.261" value="+34699680261" target="_blank">+34 699.680.261

2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:

Hi,

I tried various tools, e.g.

* SAGA reclassify
* GRASS r.mapcalc

Both work fine on smaller data sets, but fail on big data sets.

This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM
and 40GB disk free on C:, more free space on other disks where the
data resides.

I am not totally sure if it is the Bigtiff issue or some other
issue. It is just a guess that maybe I am hitting a limit with tiff.
It is not fully transparent to me what Processing is doing in the
background (what file formats with what options it creates).

It may also be an issue with SAGA or GRASS directly.

Thanks if you know anything,
Andreas

Am 2014-11-18 21:15, schrieb Giovanni Manghi:

Hi,

I am having a lot of troubles with large raster files and
processing.

The intermediate results can go >4GB. Processing stops with not so
helpful error messages.

I wonder if processing is hitting the 4GB limit. Can I force
processing
to use the BIGTIFF option or a raster format that can handle >4GB
files?

Hi Andreas, what tool specifically?

cheers

-- Giovanni --
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer [1]
 _______________________________________________
 Qgis-developer mailing list
 [hidden email]
 http://lists.osgeo.org/mailman/listinfo/qgis-developer [1]



Links:
------
[1] http://lists.osgeo.org/mailman/listinfo/qgis-developer
[2] http://www.psig.es


_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

giohappy
In reply to this post by Andreas Neumann-4
And do you have problems with GRASS (directly, not through Processing)?
In GRASS7 you can even drectly write to Geotiff without passing through the internal format (http://grass.osgeo.org/grass70/manuals/r.external.out.html).
If GDAL is built with bigtiff support there souldn't be problems...

giovanni

2014-11-19 10:44 GMT+01:00 Andreas Neumann <[hidden email]>:
Hi Giovanni,

I need to change all pixels that have the value of 249 to a value of 255. All other pixels should stay at their value.

Andreas

Am 2014-11-19 10:21, schrieb G. Allegri:
From my grayscale tiff images I want to replace one pixel value
(249) with white (255). All other pixels should stay at the value they
have.
 >
 > I came across https://github.com/chiatt/gdal_reclassify [1] - but
this reclassifies all pixels, not only selected ones.
 >
 > Maybe I could use http://www.gdal.org/gdal_calc.html [2] with a
conditional syntax?

Hi Andreas, do you need to change the value of one specific pixel
only? I mean the pixel at [x,y] index?

giovanni


 > Thanks,
 > Andreas
 >
 > Am 2014-11-19 09:10, schrieb Carlos López PSIG:
 >>
 >> Hi Andreas,
 >>
 >> I also had problems to reclassify BIGTIFF with SAGA and GRASS!
 >> Finally I used GDAL!
 >>
 >> Best regards,
 >>
 >>  [2]
 >>
 >> CARLOS LÓPEZ QUINTANILLA
 >>
 >> www.psig.es [3] [2]

 >>
 >> [hidden email]
 >> <a href="tel:%2B34%20699.680.261" value="+34699680261" target="_blank">+34 699.680.261
 >>
 >> 2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:
 >>
 >>> Hi,
 >>>
 >>> I tried various tools, e.g.
 >>>
 >>> * SAGA reclassify
 >>> * GRASS r.mapcalc
 >>>
 >>> Both work fine on smaller data sets, but fail on big data sets.
 >>>
 >>> This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM
 >>> and 40GB disk free on C:, more free space on other disks where
the
 >>> data resides.
 >>>
 >>> I am not totally sure if it is the Bigtiff issue or some other
 >>> issue. It is just a guess that maybe I am hitting a limit with
tiff.
 >>> It is not fully transparent to me what Processing is doing in the
 >>> background (what file formats with what options it creates).
 >>>
 >>> It may also be an issue with SAGA or GRASS directly.
 >>>
 >>> Thanks if you know anything,
 >>> Andreas
 >>>
 >>> Am 2014-11-18 21:15, schrieb Giovanni Manghi:
 >>>
 >>> Hi,
 >>>
 >>> I am having a lot of troubles with large raster files and
 >>> processing.
 >>>
 >>> The intermediate results can go >4GB. Processing stops with not
so
 >>> helpful error messages.
 >>>
 >>> I wonder if processing is hitting the 4GB limit. Can I force
 >>> processing
 >>> to use the BIGTIFF option or a raster format that can handle >4GB
 >>> files?
 >>>
 >>> Hi Andreas, what tool specifically?
 >>>
 >>> cheers
 >>>
 >>> -- Giovanni --
 >>> _______________________________________________
 >>> Qgis-developer mailing list
 >>> [hidden email]
 >>> http://lists.osgeo.org/mailman/listinfo/qgis-developer [4] [1]
 >>
 >>  _______________________________________________
 >>  Qgis-developer mailing list
 >>  [hidden email]
 >>  http://lists.osgeo.org/mailman/listinfo/qgis-developer [4] [1]
 >>
 >>
 >>
 >> Links:
 >> ------
 >> [1] http://lists.osgeo.org/mailman/listinfo/qgis-developer [4]
 >> [2] http://www.psig.es [3]
 >
 > _______________________________________________
 > Qgis-developer mailing list
 > [hidden email]
 > http://lists.osgeo.org/mailman/listinfo/qgis-developer [4]

Links:
------
[1] https://github.com/chiatt/gdal_reclassify
[2] http://www.gdal.org/gdal_calc.html
[3] http://www.psig.es
[4] http://lists.osgeo.org/mailman/listinfo/qgis-developer



--

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Andreas Neumann-4
Hi,

I will have to do further testing about GRASS. On Windows (OSGeo4W install) it does not work too well. It probably works better on Linux.

The gdal_calc.py solution works very fast though - glad that we came to this solution!

Andreas

On 19.11.2014 11:25, G. Allegri wrote:
And do you have problems with GRASS (directly, not through Processing)?
In GRASS7 you can even drectly write to Geotiff without passing through the internal format (http://grass.osgeo.org/grass70/manuals/r.external.out.html).
If GDAL is built with bigtiff support there souldn't be problems...

giovanni

2014-11-19 10:44 GMT+01:00 Andreas Neumann <[hidden email]>:
Hi Giovanni,

I need to change all pixels that have the value of 249 to a value of 255. All other pixels should stay at their value.

Andreas

Am 2014-11-19 10:21, schrieb G. Allegri:
From my grayscale tiff images I want to replace one pixel value
(249) with white (255). All other pixels should stay at the value they
have.
 >
 > I came across https://github.com/chiatt/gdal_reclassify [1] - but
this reclassifies all pixels, not only selected ones.
 >
 > Maybe I could use http://www.gdal.org/gdal_calc.html [2] with a
conditional syntax?

Hi Andreas, do you need to change the value of one specific pixel
only? I mean the pixel at [x,y] index?

giovanni


 > Thanks,
 > Andreas
 >
 > Am 2014-11-19 09:10, schrieb Carlos López PSIG:
 >>
 >> Hi Andreas,
 >>
 >> I also had problems to reclassify BIGTIFF with SAGA and GRASS!
 >> Finally I used GDAL!
 >>
 >> Best regards,
 >>
 >>  [2]
 >>
 >> CARLOS LÓPEZ QUINTANILLA
 >>
 >> www.psig.es [3] [2]

 >>
 >> [hidden email]
 >> <a moz-do-not-send="true" href="tel:%2B34%20699.680.261" value="+34699680261" target="_blank">+34 699.680.261
 >>
 >> 2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:
 >>
 >>> Hi,
 >>>
 >>> I tried various tools, e.g.
 >>>
 >>> * SAGA reclassify
 >>> * GRASS r.mapcalc
 >>>
 >>> Both work fine on smaller data sets, but fail on big data sets.
 >>>
 >>> This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM
 >>> and 40GB disk free on C:, more free space on other disks where
the
 >>> data resides.
 >>>
 >>> I am not totally sure if it is the Bigtiff issue or some other
 >>> issue. It is just a guess that maybe I am hitting a limit with
tiff.
 >>> It is not fully transparent to me what Processing is doing in the
 >>> background (what file formats with what options it creates).
 >>>
 >>> It may also be an issue with SAGA or GRASS directly.
 >>>
 >>> Thanks if you know anything,
 >>> Andreas
 >>>
 >>> Am 2014-11-18 21:15, schrieb Giovanni Manghi:
 >>>
 >>> Hi,
 >>>
 >>> I am having a lot of troubles with large raster files and
 >>> processing.
 >>>
 >>> The intermediate results can go >4GB. Processing stops with not
so
 >>> helpful error messages.
 >>>
 >>> I wonder if processing is hitting the 4GB limit. Can I force
 >>> processing
 >>> to use the BIGTIFF option or a raster format that can handle >4GB
 >>> files?
 >>>
 >>> Hi Andreas, what tool specifically?
 >>>
 >>> cheers
 >>>
 >>> -- Giovanni --
 >>> _______________________________________________
 >>> Qgis-developer mailing list
 >>> [hidden email]
 >>> http://lists.osgeo.org/mailman/listinfo/qgis-developer [4] [1]
 >>
 >>  _______________________________________________
 >>  Qgis-developer mailing list
 >>  [hidden email]
 >>  http://lists.osgeo.org/mailman/listinfo/qgis-developer [4] [1]
 >>
 >>
 >>
 >> Links:
 >> ------
 >> [1] http://lists.osgeo.org/mailman/listinfo/qgis-developer [4]
 >> [2] http://www.psig.es [3]
 >
 > _______________________________________________
 > Qgis-developer mailing list
 > [hidden email]
 > http://lists.osgeo.org/mailman/listinfo/qgis-developer [4]

Links:
------
[1] https://github.com/chiatt/gdal_reclassify
[2] http://www.gdal.org/gdal_calc.html
[3] http://www.psig.es
[4] http://lists.osgeo.org/mailman/listinfo/qgis-developer



--


_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

giohappy
Nice to know that gdal_calc works fine ;)
I work a lot with gdal + numpy, even for custom tools, and I admit they work very well together!


2014-11-19 11:41 GMT+01:00 Andreas Neumann <[hidden email]>:
Hi,

I will have to do further testing about GRASS. On Windows (OSGeo4W install) it does not work too well. It probably works better on Linux.

The gdal_calc.py solution works very fast though - glad that we came to this solution!

Andreas


On 19.11.2014 11:25, G. Allegri wrote:
And do you have problems with GRASS (directly, not through Processing)?
In GRASS7 you can even drectly write to Geotiff without passing through the internal format (http://grass.osgeo.org/grass70/manuals/r.external.out.html).
If GDAL is built with bigtiff support there souldn't be problems...

giovanni

2014-11-19 10:44 GMT+01:00 Andreas Neumann <[hidden email]>:
Hi Giovanni,

I need to change all pixels that have the value of 249 to a value of 255. All other pixels should stay at their value.

Andreas

Am 2014-11-19 10:21, schrieb G. Allegri:
From my grayscale tiff images I want to replace one pixel value
(249) with white (255). All other pixels should stay at the value they
have.
 >
 > I came across https://github.com/chiatt/gdal_reclassify [1] - but
this reclassifies all pixels, not only selected ones.
 >
 > Maybe I could use http://www.gdal.org/gdal_calc.html [2] with a
conditional syntax?

Hi Andreas, do you need to change the value of one specific pixel
only? I mean the pixel at [x,y] index?

giovanni


 > Thanks,
 > Andreas
 >
 > Am 2014-11-19 09:10, schrieb Carlos López PSIG:
 >>
 >> Hi Andreas,
 >>
 >> I also had problems to reclassify BIGTIFF with SAGA and GRASS!
 >> Finally I used GDAL!
 >>
 >> Best regards,
 >>
 >>  [2]
 >>
 >> CARLOS LÓPEZ QUINTANILLA
 >>
 >> www.psig.es [3] [2]

 >>
 >> [hidden email]
 >> <a href="tel:%2B34%20699.680.261" value="+34699680261" target="_blank">+34 699.680.261
 >>
 >> 2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:
 >>
 >>> Hi,
 >>>
 >>> I tried various tools, e.g.
 >>>
 >>> * SAGA reclassify
 >>> * GRASS r.mapcalc
 >>>
 >>> Both work fine on smaller data sets, but fail on big data sets.
 >>>
 >>> This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM
 >>> and 40GB disk free on C:, more free space on other disks where
the
 >>> data resides.
 >>>
 >>> I am not totally sure if it is the Bigtiff issue or some other
 >>> issue. It is just a guess that maybe I am hitting a limit with
tiff.
 >>> It is not fully transparent to me what Processing is doing in the
 >>> background (what file formats with what options it creates).
 >>>
 >>> It may also be an issue with SAGA or GRASS directly.
 >>>
 >>> Thanks if you know anything,
 >>> Andreas
 >>>
 >>> Am 2014-11-18 21:15, schrieb Giovanni Manghi:
 >>>
 >>> Hi,
 >>>
 >>> I am having a lot of troubles with large raster files and
 >>> processing.
 >>>
 >>> The intermediate results can go >4GB. Processing stops with not
so
 >>> helpful error messages.
 >>>
 >>> I wonder if processing is hitting the 4GB limit. Can I force
 >>> processing
 >>> to use the BIGTIFF option or a raster format that can handle >4GB
 >>> files?
 >>>
 >>> Hi Andreas, what tool specifically?
 >>>
 >>> cheers
 >>>
 >>> -- Giovanni --
 >>> _______________________________________________
 >>> Qgis-developer mailing list
 >>> [hidden email]
 >>> http://lists.osgeo.org/mailman/listinfo/qgis-developer [4] [1]
 >>
 >>  _______________________________________________
 >>  Qgis-developer mailing list
 >>  [hidden email]
 >>  http://lists.osgeo.org/mailman/listinfo/qgis-developer [4] [1]
 >>
 >>
 >>
 >> Links:
 >> ------
 >> [1] http://lists.osgeo.org/mailman/listinfo/qgis-developer [4]
 >> [2] http://www.psig.es [3]
 >
 > _______________________________________________
 > Qgis-developer mailing list
 > [hidden email]
 > http://lists.osgeo.org/mailman/listinfo/qgis-developer [4]

Links:
------
[1] https://github.com/chiatt/gdal_reclassify
[2] http://www.gdal.org/gdal_calc.html
[3] http://www.psig.es
[4] http://lists.osgeo.org/mailman/listinfo/qgis-developer



--




--

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

pcav
In reply to this post by Andreas Neumann-4
Il 19/11/2014 11:41, Andreas Neumann ha scritto:

> I will have to do further testing about GRASS. On Windows (OSGeo4W install) it does
> not work too well. It probably works better on Linux.

Why so? It shouldn't, at lest not grass64.
IMHO dealing with very large rasters is one of the strengths of the good old grass.
Also OTB way of handling large files is very interesting.
All the best.

--
Paolo Cavallini - www.faunalia.eu
Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Andreas Neumann-4
In reply to this post by Carlos López PSIG
Hi,

I was able to do my color replacements with gdal_calc.py. It works fast
and efficient.

Here is my syntax:

gdal_calc.py -A image.tif --outfile=image_output.tif
--calc="(A==249)*255+(A!=249)*A"

Syntax looks similar to the QGIS raster calculator - but as opposed to
the QGIS raster calculator it actually works!

I will still continue to find out about SAGA. Maybe directly converting
from TIFF to SAGA before processing files in SAGA.

I believe that the QGIS raster calculator would need some developer love
- to make it more robust and easier to use ;-)

I am also pretty sure that the Processing integration of GRASS/SAGA does
not work well with big files - at least on Windows and when TIFF files
are used.

Thanks to all who helped!
Andreas


Am 2014-11-19 11:01, schrieb Carlos López PSIG:

> Hi Andreas,
>
> Yes, you can use gdal_calc.py
> Certainly you need to perform several processes, but are very fast!
>
> Be careful with bands!
> Perhaps you will want to separate them.
>
> Ciao,
>
>  [3]
>
> CARLOS LÓPEZ QUINTANILLA
>
> www.psig.es [3]
> [hidden email]
> +34 699.680.261
>
> 2014-11-19 9:26 GMT+01:00 Andreas Neumann <[hidden email]>:
>
>> Hi Carlos,
>>
>> GDAL would be my first choice - because it easy to automate.
>> However, I don't know how can do my task with GDAL.
>>
>> From my grayscale tiff images I want to replace one pixel value
>> (249) with white (255). All other pixels should stay at the value
>> they have.
>>
>> I came across https://github.com/chiatt/gdal_reclassify [1] - but
>> this reclassifies all pixels, not only selected ones.
>>
>> Maybe I could use http://www.gdal.org/gdal_calc.html [2] with a
>> conditional syntax?
>>
>> Thanks,
>> Andreas
>>
>> Am 2014-11-19 09:10, schrieb Carlos López PSIG:
>> Hi Andreas,
>>
>> I also had problems to reclassify BIGTIFF with SAGA and GRASS!
>> Finally I used GDAL!
>>
>> Best regards,
>>
>>  [2]
>>
>> CARLOS LÓPEZ QUINTANILLA
>>
>> www.psig.es [3] [2]
>>
>> [hidden email]
>> +34 699.680.261 [4]
>>
>> 2014-11-19 9:05 GMT+01:00 Andreas Neumann <[hidden email]>:
>>
>> Hi,
>>
>> I tried various tools, e.g.
>>
>> * SAGA reclassify
>> * GRASS r.mapcalc
>>
>> Both work fine on smaller data sets, but fail on big data sets.
>>
>> This is on Win7 64bit, with the OSGeo4W installer. I have 8gb RAM
>> and 40GB disk free on C:, more free space on other disks where the
>> data resides.
>>
>> I am not totally sure if it is the Bigtiff issue or some other
>> issue. It is just a guess that maybe I am hitting a limit with
>> tiff.
>> It is not fully transparent to me what Processing is doing in the
>> background (what file formats with what options it creates).
>>
>> It may also be an issue with SAGA or GRASS directly.
>>
>> Thanks if you know anything,
>> Andreas
>>
>> Am 2014-11-18 21:15, schrieb Giovanni Manghi:
>>
>> Hi,
>>
>> I am having a lot of troubles with large raster files and
>> processing.
>>
>> The intermediate results can go >4GB. Processing stops with not so
>> helpful error messages.
>>
>> I wonder if processing is hitting the 4GB limit. Can I force
>> processing
>> to use the BIGTIFF option or a raster format that can handle >4GB
>> files?
>>
>> Hi Andreas, what tool specifically?
>>
>> cheers
>>
>> -- Giovanni --
>> _______________________________________________
>> Qgis-developer mailing list
>> [hidden email]
>> http://lists.osgeo.org/mailman/listinfo/qgis-developer [5] [1]
>>  _______________________________________________
>>  Qgis-developer mailing list
>>  [hidden email]
>>  http://lists.osgeo.org/mailman/listinfo/qgis-developer [5] [1]
>>
>> Links:
>> ------
>> [1] http://lists.osgeo.org/mailman/listinfo/qgis-developer [5]
>> [2] http://www.psig.es [3]
>
>
>
> Links:
> ------
> [1] https://github.com/chiatt/gdal_reclassify
> [2] http://www.gdal.org/gdal_calc.html
> [3] http://www.psig.es
> [4] tel:%2B34%20699.680.261
> [5] http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Giovanni Manghi-2
In reply to this post by Andreas Neumann-4
Hi Andreas,

> It may also be an issue with SAGA or GRASS directly.

I have also tested a very large raster (8gb) and SAGA seems to choke
also when used natively from the cli interface.

grass of course does the job, especially when you use the grass data
model from the cli, native grass gui or qgis plugin. Inside processing
large raster suffer from the overhead that is needed to import/export
the raster to, and out of, a temp location.

gdal does the reclass job very quickly and I should be able to add the
tool to processing.

I agree anyway that it too bad that the qgis raster calculator (and
georeferencer) are drawing so little attention. Maybe there isn't much
people that uses them?

cheers!

-- G --
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

pcav
Il 21/11/2014 15:13, Giovanni Manghi ha scritto:

> grass of course does the job, especially when you use the grass data
> model from the cli, native grass gui or qgis plugin. Inside processing
> large raster suffer from the overhead that is needed to import/export
> the raster to, and out of, a temp location.

AFAIK Processing does not import the raster, but uses it as an external data source
(if not, it should).

> I agree anyway that it too bad that the qgis raster calculator (and
> georeferencer) are drawing so little attention. Maybe there isn't much
> people that uses them?

quite possible; however, maintaining multiple calcs seems more of a burden than an
advantage.

all the best.
--
Paolo Cavallini - www.faunalia.eu
Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

Giovanni Manghi-2
In reply to this post by Andreas Neumann-4
> AFAIK Processing does not import the raster, but uses it as an external data source
> (if not, it should).

yes you are right, it uses r.external.



>> I agree anyway that it too bad that the qgis raster calculator (and
>> georeferencer) are drawing so little attention. Maybe there isn't much
>> people that uses them?
>
> quite possible; however, maintaining multiple calcs seems more of a burden than an
> advantage.


yes I understand, but we have a clear issue. The qgis raster calc is
broken and when it works it need a very cumbersome logic (especially
for common tasks like reclassifying). GRASS r.reclass works fine, but
it does not allow to enter interactively the reclass rules (that can
be an advantage in some situations, but usually for quick
reclassifications is not), so adding gdal raster calc seems anyway a
good idea.

cheers!
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Processing with raster files > 4GB

pcav
Hi all.

Il 22/11/2014 14:28, Giovanni Manghi ha scritto:
> yes I understand, but we have a clear issue. The qgis raster calc is
> broken and when it works it need a very cumbersome logic (especially
> for common tasks like reclassifying). GRASS r.reclass works fine, but
> it does not allow to enter interactively the reclass rules (that can
> be an advantage in some situations, but usually for quick
> reclassifications is not), so adding gdal raster calc seems anyway a
> good idea.

sorry I was unclear: I meant I do not see the point in maintaining a qgis raster
calc, given the problems you refer, and the availability of several alternatives
(also the saga one).
I agree that adding gdal raster calc is a plus, especially because it requires
minimal effort compared to fixing and maintaining qgis one.
All the best.

--
Paolo Cavallini - www.faunalia.eu
Corsi QGIS e PostGIS: http://www.faunalia.eu/training.html
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer