osgeo4w portability

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

osgeo4w portability

Matt Wilkie
Hi Folks,

A discussion started off-list about a portable Osgeo4W (here).

Jurgen observed that "most of the batch files are "portable", but we also have configuration file templates* that are modified in postinstall using textreplace. That's essentially why it exists.

BTW another problem with portability are the msvcrt packages - that need to be installed." [ *c.f. the batch shell template ]

I've used o4w portably for a long time by simply copying C:\Osgeo4W tree to a usb stick and I've not run into problems, but I mostly use just the gdal command line utils. I thought it would be good to collect and/or itemize the particular packages or configurations that break portability. We could then look and see if it's unavoidable or there might be some reasonably inexpensive way of changing that.

This this thread; what doesn't work portably right now?

Matt

_______________________________________________
osgeo4w-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/osgeo4w-dev
Reply | Threaded
Open this post in threaded view
|

Re: osgeo4w portability

Alex Mandel-2
On 02/05/2015 02:39 PM, Matt Wilkie wrote:

> Hi Folks,
>
> A discussion started off-list about a portable Osgeo4W (here
> <https://github.com/maphew/apt/issues/10#issuecomment-73048724>).
>
> Jurgen observed that *"most of the batch files are "portable", but we also
> have configuration file templates* that are modified in postinstall using
> textreplace. That's essentially why it exists. *
>
> *BTW another problem with portability are the msvcrt packages - that need
> to be installed." *[ *c.f. the batch shell template
> <http://trac.osgeo.org/osgeo4w/wiki/PackagingInstructions#Applicationbatchfiletemplate>
> ]
> I've used o4w portably for a long time by simply copying C:\Osgeo4W tree to
> a usb stick and I've not run into problems, but I mostly use just the gdal
> command line utils. I thought it would be good to collect and/or itemize
> the particular packages or configurations that break portability. We could
> then look and see if it's unavoidable or there might be some reasonably
> inexpensive way of changing that.
>
> This this thread; what doesn't work portably right now?
>
> Matt
>

You should talk to J. Cook this is extremely similar to
http://www.archaeogeek.com/blog/2014/01/20/portable-gis-v4/

Thanks,
Alex

_______________________________________________
osgeo4w-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/osgeo4w-dev
Reply | Threaded
Open this post in threaded view
|

Re: osgeo4w portability

mhw-at-yg
> You should talk to J. Cook this is extremely similar to http://www.archaeogeek.com/blog/2014/01/20/portable-gis-v4/

Thanks Alex, that's a new project to me, looks interesting.

Matt
_______________________________________________
osgeo4w-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/osgeo4w-dev
-Matt
Reply | Threaded
Open this post in threaded view
|

osgeo4w portability

Müller, Andreas, Dr. phil.
In reply to this post by Matt Wilkie
Hi Matt and Alex and all readers,
First: I hope this post fits in the right place, because i had no mail to respond to...

I asked Matt at github, https://github.com/maphew/apt/issues/10#issuecomment-73048724, because i wanted his apt.py to work on a portable install for OSGeo4W packages. So he opened this thread. Thank you for that!

To have a portable OSGeo4W, you could easily copy from an installed version. Much works, but it is not really portable. For eg. you should start QGIS with option configpath, like --configpath "%HOME%\.qgis". That prevents QGIS to write to the Windows-Registry.

There are attempts to build a portable gqis, osgeo4w Version, but, like the one mentioned by Alex, http://www.archaeogeek.com/blog/2014/01/20/portable-gis-v4/, it lacks, as far as i know, a possibility to choose and update packages. That's the point, where apt.py, as a package manager, joins the game! (because i'm more comfortable with python than with perl, thanks for the code of "creatensis.pl" by jef-n on github)

Unfortunately nearly all packages go through the postinstall stage of installation. Here, bat files in etc/postinstall are called, mainly to replace patterns like @osgeo4w@ to hard codes paths, which we don't want in a portable install. One tool to do that is textreplace, which turns @osgeo4w@ into the string stored in the environment variable %OSGEO4W_ROOT%.

For example apache:
in etc/postinstall we find apache.bat and inside
textreplace -sf apache/conf/httpd.conf -df apache/conf/httpd.conf -map @osgeo4w@ %OSGEO4W_ROOT%
which changes the pattern in http.conf.
A portable version would be
textreplace -sf apache/conf/httpd.conf -df apache/conf/httpd.conf -map @osgeo4w@ ${OSGEO4W_ROOT}
now it's up to apache to find the path through the environment variable.

A lot of packages make use of textreplace, i didn't check them all. And there are cases, where something different is wanted to happen, where software must be installed to the System, like Visual-C Re-distributables.

What i'm asking for, is there a way to deal with postinstall-issues, even if i can't predict which package is to be installed.

One solution would be to delegate that through the Packaking-instructions (https://trac.osgeo.org/osgeo4w/wiki/PackagingInstructions). But is this realistic?


if someone interested or willing to help, i can show the cases i found, during my version of a "portable installer".

Best Regards,
Andreas

Dr. Andreas Müller -_- Universität Trier - FB VI - Kartographie, [hidden email]
54286 Trier, Behringstr. 29, F 191, ++496512014576
_______________________________________________
osgeo4w-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/osgeo4w-dev
Reply | Threaded
Open this post in threaded view
|

Re: osgeo4w portability

Matt Wilkie
Andreas,

Thanks for the specific examples. That's exactly what I thought of collecting in this thread. Even if there's no immediate solution or response forthcoming I encourage more of them (from anyone).

I'm only able to spend a few stolen hours here and there on apt and related packaging efforts, so progress from
me comes in fits and starts, but it is something I'll be continuing to move forward. ;-)

cheers,

matt

On Mon, Feb 9, 2015 at 4:34 AM, Müller, Andreas, Dr. phil. <[hidden email]> wrote:
Hi Matt and Alex and all readers,
First: I hope this post fits in the right place, because i had no mail to respond to...

I asked Matt at github, https://github.com/maphew/apt/issues/10#issuecomment-73048724, because i wanted his apt.py to work on a portable install for OSGeo4W packages. So he opened this thread. Thank you for that!

To have a portable OSGeo4W, you could easily copy from an installed version. Much works, but it is not really portable. For eg. you should start QGIS with option configpath, like --configpath "%HOME%\.qgis". That prevents QGIS to write to the Windows-Registry.

There are attempts to build a portable gqis, osgeo4w Version, but, like the one mentioned by Alex, http://www.archaeogeek.com/blog/2014/01/20/portable-gis-v4/, it lacks, as far as i know, a possibility to choose and update packages. That's the point, where apt.py, as a package manager, joins the game! (because i'm more comfortable with python than with perl, thanks for the code of "creatensis.pl" by jef-n on github)

Unfortunately nearly all packages go through the postinstall stage of installation. Here, bat files in etc/postinstall are called, mainly to replace patterns like @osgeo4w@ to hard codes paths, which we don't want in a portable install. One tool to do that is textreplace, which turns @osgeo4w@ into the string stored in the environment variable %OSGEO4W_ROOT%.

For example apache:
in etc/postinstall we find apache.bat and inside
textreplace -sf apache/conf/httpd.conf -df apache/conf/httpd.conf -map @osgeo4w@ %OSGEO4W_ROOT%
which changes the pattern in http.conf.
A portable version would be
textreplace -sf apache/conf/httpd.conf -df apache/conf/httpd.conf -map @osgeo4w@ ${OSGEO4W_ROOT}
now it's up to apache to find the path through the environment variable.

A lot of packages make use of textreplace, i didn't check them all. And there are cases, where something different is wanted to happen, where software must be installed to the System, like Visual-C Re-distributables.

What i'm asking for, is there a way to deal with postinstall-issues, even if i can't predict which package is to be installed.

One solution would be to delegate that through the Packaking-instructions (https://trac.osgeo.org/osgeo4w/wiki/PackagingInstructions). But is this realistic?


if someone interested or willing to help, i can show the cases i found, during my version of a "portable installer".

Best Regards,
Andreas

Dr. Andreas Müller -_- Universität Trier - FB VI - Kartographie, [hidden email]
54286 Trier, Behringstr. 29, F 191, <a href="tel:%2B%2B496512014576" value="+496512014576">++496512014576
_______________________________________________
osgeo4w-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/osgeo4w-dev


_______________________________________________
osgeo4w-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/osgeo4w-dev
Reply | Threaded
Open this post in threaded view
|

Re: osgeo4w portability

Müller, Andreas, Dr. phil.
Me again,
today i will focus on OSGEO4W-Apps and how to get them portable. With portable i mean to have an computer independent install eg. on an USB-flash drive.
So here are the apps, which are not completely portable because postinstall patches files with full pathnames.

gdal-(16,17): gdal-config, a bourne shell script, which i use like this:
--
#!/bin/sh/
CONFIG_LIBS="$osgeo4w_msys/lib/gdal_i.lib"
CONFIG_DEP_LIBS="-L$osgeo4w_msys/lib -lpng -lz"
CONFIG_PREFIX="$osgeo4w_msys"
CONFIG_CFLAGS="-I$osgeo4w_msys/include"
....
--

gpsbabel: has a qt.conf with windows like paths and windows like ini file format. Changed it to:
--
[paths]
Prefix=%OSGEO4W_ROOT%\\apps\\qt4
Binaries=%OSGEO4W_ROOT%\\bin
Libraries=%OSGEO4W_ROOT%\\lib
Headers=%OSGEO4W_ROOT%\\include\\qt4
--
BUT: i don't know if the win environment is interpretated?!

Python2(6,7)
I always mentioned python-PyQt4 with sipconfig and pyqtconfig.

qgis:
has a qgis.reg, which could be an empty file. Better use --configpath "%HOME%\.qgis" in qgis.bat

Qt4:
has a qmake.conf - seems to be Unix-like makefile so i changed it as follows:
$$OSGEO4W_ROOT/apps/Qt4/mkspecs/win32-msvc2010

What do you think ?


further worth mentioned:
batch files in osgeo4w/bin:
beegis.bat: why not using env variable OSGEO4W_ROOT throughout the whole file?
-> change first ine: @call %OSGEO4W_ROOT%\bin\o4w_env.bat

designer-qgis-dev.bat: well done, it uses %~dp0 to point to right directory:
call "%~dp0\o4w_env.bat"

gpsbaelfe.bat: Don't hardcode OSGEO4W_ROOT, but call o4w_env.bat with %~dp0
same with: grassXX.bat, liblas, monteverdi, nettools, pyuic, setup and udig

also we have a qt.conf here and qgis.bat (and browser, dev etc.) which could easily be edited as i suggested above!

Thank you very much for reading,
best regards,
Andreas
--
Dr. Andreas Müller -_- Universität Trier - FB VI - Kartographie, [hidden email]

54286 Trier, Behringstr. 29, F 191, ++496512014576


Von: Müller, Andreas, Dr. phil.
Gesendet: Mittwoch, 18. Februar 2015 12:30
An: Matt Wilkie
Betreff: AW: [osgeo4w-dev] osgeo4w portability

Matt,
after i understood what exactly apt does and after your last contributions to it, i am happy with your code! I made a full download of all packages and changed all "postinstall infected" files to use them as a reference for future changes and new packages. My plan is to break the installation, copy those files, if nothing changed (i played around with difflib: http://pymotw.com/2/difflib/) and continue install with registering in setup.ini and installed.db
There are always pitfalls, i can't foresee, in particular that i'm not able to test all changes i made, due to lacks of experience and time. As my main goal is to give the portable osgeo4w to my students, at least i can misuse them to serve as "guinea pigs" ;-)

I talked about "cases" of non-portability, here is one more - it was hard to find: PyQt4 has config-templates with hard coded paths in pyqtconfig.py and sipconfig.py! I am a little surprised that it works however? But there was also a solution on http://www.mantidproject.org/Building_Windows_Dependencies, as i reported here: https://hub.qgis.org/issues/11430.

Oh, and for python programmers: i always missed "idle" in OSGEO4W_ROOT\bin:
rem idle.bat
@echo off
call %OSGEO4W_ROOT%\bin\o4w_env.bat
start "IDLE" /D%OSGEO4W_ROOT%\apps\python27\lib\idlelib /B pythonw idle.pyw %*

regards,
Andreas

Dr. Andreas Müller -_- Universität Trier - FB VI - Kartographie, [hidden email]

54286 Trier, Behringstr. 29, F 191, ++496512014576


Von: Matt Wilkie [[hidden email]]
Gesendet: Mittwoch, 18. Februar 2015 09:03
An: Müller, Andreas, Dr. phil.
Cc: [hidden email]
Betreff: Re: [osgeo4w-dev] osgeo4w portability

Andreas,

Thanks for the specific examples. That's exactly what I thought of collecting in this thread. Even if there's no immediate solution or response forthcoming I encourage more of them (from anyone).

I'm only able to spend a few stolen hours here and there on apt and related packaging efforts, so progress from
me comes in fits and starts, but it is something I'll be continuing to move forward. ;-)

cheers,

matt

On Mon, Feb 9, 2015 at 4:34 AM, Müller, Andreas, Dr. phil. <[hidden email]> wrote:
Hi Matt and Alex and all readers,
First: I hope this post fits in the right place, because i had no mail to respond to...

I asked Matt at github, https://github.com/maphew/apt/issues/10#issuecomment-73048724, because i wanted his apt.py to work on a portable install for OSGeo4W packages. So he opened this thread. Thank you for that!

To have a portable OSGeo4W, you could easily copy from an installed version. Much works, but it is not really portable. For eg. you should start QGIS with option configpath, like --configpath "%HOME%\.qgis". That prevents QGIS to write to the Windows-Registry.

There are attempts to build a portable gqis, osgeo4w Version, but, like the one mentioned by Alex, http://www.archaeogeek.com/blog/2014/01/20/portable-gis-v4/, it lacks, as far as i know, a possibility to choose and update packages. That's the point, where apt.py, as a package manager, joins the game! (because i'm more comfortable with python than with perl, thanks for the code of "creatensis.pl" by jef-n on github)

Unfortunately nearly all packages go through the postinstall stage of installation. Here, bat files in etc/postinstall are called, mainly to replace patterns like @osgeo4w@ to hard codes paths, which we don't want in a portable install. One tool to do that is textreplace, which turns @osgeo4w@ into the string stored in the environment variable %OSGEO4W_ROOT%.

For example apache:
in etc/postinstall we find apache.bat and inside
textreplace -sf apache/conf/httpd.conf -df apache/conf/httpd.conf -map @osgeo4w@ %OSGEO4W_ROOT%
which changes the pattern in http.conf.
A portable version would be
textreplace -sf apache/conf/httpd.conf -df apache/conf/httpd.conf -map @osgeo4w@ ${OSGEO4W_ROOT}
now it's up to apache to find the path through the environment variable.

A lot of packages make use of textreplace, i didn't check them all. And there are cases, where something different is wanted to happen, where software must be installed to the System, like Visual-C Re-distributables.

What i'm asking for, is there a way to deal with postinstall-issues, even if i can't predict which package is to be installed.

One solution would be to delegate that through the Packaking-instructions (https://trac.osgeo.org/osgeo4w/wiki/PackagingInstructions). But is this realistic?


if someone interested or willing to help, i can show the cases i found, during my version of a "portable installer".

Best Regards,
Andreas

Dr. Andreas Müller -_- Universität Trier - FB VI - Kartographie, [hidden email]
54286 Trier, Behringstr. 29, F 191, <a href="tel:%2B%2B496512014576" value="&#43;496512014576" target="_blank"> ++496512014576
_______________________________________________
osgeo4w-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/osgeo4w-dev


_______________________________________________
osgeo4w-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/osgeo4w-dev
Reply | Threaded
Open this post in threaded view
|

Re: osgeo4w portability

Müller, Andreas, Dr. phil.
finished a rough installer for a portable install, and created a repository on github: https://github.com/amr66/osgeo4w-portable
Right now files aren't complete, but the main basis is on-line. If anyone is interested, let me know. So I have questions:
My installation needs additional software - they are a way hard coded inside my work. Is it common to upload such files? As far as i know there are no restrictions through licenses...
I included postgresql/postgis zip-files, PyScripter executables, the author of the free software PStart i use, doesn't allow distributing, but linking to his site. I'll have to find a solution...

Dr. Andreas Müller -_- Universität Trier - FB VI - Kartographie, [hidden email]

54286 Trier, Behringstr. 29, F 191, ++496512014576



_______________________________________________
osgeo4w-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/osgeo4w-dev