[QGIS-Developer] External python package dependency in plugins

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[QGIS-Developer] External python package dependency in plugins

shiva reddy
Hello,
I tried to install automatically during my pluign (QRealTime) installation by calling:

 subprocess.call(['python','-m','install','pyxform','--user'])

It worked many times in windows and it fails sometime.
It don't work in OS X and I have not tested in ubuntu.

What is the best practice for such situation?


Thanks & Regards
Shiva Reddy K.
Scientist/Engineer 'SD'
Indian Institute of Remote Sensing,
Indian Space Research Organisation
Department of Space
4-Kalidas Road
Dehradun
mobile: 7017035414

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

shiva reddy
minor correction:

subprocess.call(['python','-m','pip','install','pyxform','--user'])
Thanks & Regards
Shiva Reddy K.
Scientist/Engineer 'SC'
Indian Institute of Remote Sensing,
Indian Space Research Organisation
Department of Space
4-Kalidas Road
Dehradun
mobile: 8791806093


On Sat, Jul 7, 2018 at 11:40 AM shiva reddy <[hidden email]> wrote:
Hello,
I tried to install automatically during my pluign (QRealTime) installation by calling:

 subprocess.call(['python','-m','install','pyxform','--user'])

It worked many times in windows and it fails sometime.
It don't work in OS X and I have not tested in ubuntu.

What is the best practice for such situation?


Thanks & Regards
Shiva Reddy K.
Scientist/Engineer 'SD'
Indian Institute of Remote Sensing,
Indian Space Research Organisation
Department of Space
4-Kalidas Road
Dehradun
mobile: 7017035414

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

Richard Duivenvoorde
On 07/07/2018 08:11 AM, shiva reddy wrote:

>     Hello,
>     I tried to install automatically during my pluign (QRealTime)
>     installation by calling:
>
>      subprocess.call(['python','-m','install','pyxform','--user'])
>
>     It worked many times in windows and it fails sometime.
>     It don't work in OS X and I have not tested in ubuntu.
>
>     What is the best practice for such situation?

Hi, I tried your lines here on Debian 9.4/testing/ in a python3 console:

>>> import subprocess
>>> subprocess.call(['python','-m','install','pyxform','--user'])
/usr/bin/python: No module named install
1
>>> subprocess.call(['python','-m','pip','install','pyxform','--user'])
/usr/bin/python: No module named pip
1

Ah, but with Debian we have python and python 3, so I tried:
subprocess.call(['python3','-m','pip','install','pyxform','--user'])
...
Installing collected packages: argparse, linecache2, traceback2, six,
unittest2, formencode, unicodecsv, xlrd, pyxform
Successfully installed argparse-1.4.0 formencode-1.3.1 linecache2-1.0.0
pyxform-0.11.3 six-1.11.0 traceback2-1.4.0 unicodecsv-0.14.1
unittest2-1.1.0 xlrd-1.1.0
0
Works! \o/
Not sure if (from within python) you can find out what the name of the
python executable is...

For QGIS2 we earlier recommended to just add the full module into your
plugin package. (if possible, aka python-only module).

There has also been discussion about using pip for python modules, by
adding the 'pip install line' as a separate metadata.txt line.
In that way we could either let users install it (in their global or
user env), or maybe later do this programatically.

There are questions/discussions/options:
- how to handle clashing specific versions of the same module between
different plugins
- where to install modules: installation system wide, in one QGIS
virtual environment (venv) or in a venv per plugin
- security questions (mostly a Linux discussion: should a distribution
rely on venv's or only install distribution modules)... etc

Your idea is (to me) new :-)
If working platform independently and full proof, I think it is an
interesting option!

Now with QGIS3 we have true separated profiles, we an idea would be to
add a virtual environment PER profile, so all plugins can install
modules in that environment (using your trick?), and in case of a clash
can then create 2 profiles for those plugins/modules.
Not sure though this is feasible for all OS's. I think it will be
easiest on Linux...

Others?

Regards,

Richard Duivenvoorde
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

ghtmtt
Hi,

thanks for raining this issue and to provide this beautiful solution.

Actually having plugin with external python lib dependencies was/is
tricky especially for windows system (in my experience)

> Now with QGIS3 we have true separated profiles, we an idea would be to
> add a virtual environment PER profile, so all plugins can install
> modules in that environment (using your trick?), and in case of a clash
> can then create 2 profiles for those plugins/modules.
> Not sure though this is feasible for all OS's. I think it will be
> easiest on Linux...

that's also a good idea!

Cheers

Matteo
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

shiva reddy
In reply to this post by Richard Duivenvoorde
Thanks,
In OS X following works:
subprocess.call(['pip3','install','pyxform'])

Does same works  works for Ubuntu/linux also?


for windows, I am assuming 'python' refers to python 3 inside QGIS3.


regards,

Thanks & Regards
Shiva Reddy K.
Scientist/Engineer 'SC'
Indian Institute of Remote Sensing,
Indian Space Research Organisation
Department of Space
4-Kalidas Road
Dehradun
mobile: 8791806093


On Sat, Jul 7, 2018 at 3:15 PM Richard Duivenvoorde <[hidden email]> wrote:
On 07/07/2018 08:11 AM, shiva reddy wrote:
>     Hello,
>     I tried to install automatically during my pluign (QRealTime)
>     installation by calling:
>
>      subprocess.call(['python','-m','install','pyxform','--user'])
>
>     It worked many times in windows and it fails sometime.
>     It don't work in OS X and I have not tested in ubuntu.
>
>     What is the best practice for such situation?

Hi, I tried your lines here on Debian 9.4/testing/ in a python3 console:

>>> import subprocess
>>> subprocess.call(['python','-m','install','pyxform','--user'])
/usr/bin/python: No module named install
1
>>> subprocess.call(['python','-m','pip','install','pyxform','--user'])
/usr/bin/python: No module named pip
1

Ah, but with Debian we have python and python 3, so I tried:
subprocess.call(['python3','-m','pip','install','pyxform','--user'])
...
Installing collected packages: argparse, linecache2, traceback2, six,
unittest2, formencode, unicodecsv, xlrd, pyxform
Successfully installed argparse-1.4.0 formencode-1.3.1 linecache2-1.0.0
pyxform-0.11.3 six-1.11.0 traceback2-1.4.0 unicodecsv-0.14.1
unittest2-1.1.0 xlrd-1.1.0
0
Works! \o/
Not sure if (from within python) you can find out what the name of the
python executable is...

For QGIS2 we earlier recommended to just add the full module into your
plugin package. (if possible, aka python-only module).

There has also been discussion about using pip for python modules, by
adding the 'pip install line' as a separate metadata.txt line.
In that way we could either let users install it (in their global or
user env), or maybe later do this programatically.

There are questions/discussions/options:
- how to handle clashing specific versions of the same module between
different plugins
- where to install modules: installation system wide, in one QGIS
virtual environment (venv) or in a venv per plugin
- security questions (mostly a Linux discussion: should a distribution
rely on venv's or only install distribution modules)... etc

Your idea is (to me) new :-)
If working platform independently and full proof, I think it is an
interesting option!

Now with QGIS3 we have true separated profiles, we an idea would be to
add a virtual environment PER profile, so all plugins can install
modules in that environment (using your trick?), and in case of a clash
can then create 2 profiles for those plugins/modules.
Not sure though this is feasible for all OS's. I think it will be
easiest on Linux...

Others?

Regards,

Richard Duivenvoorde

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

Richard Duivenvoorde
On 07/07/2018 01:26 PM, shiva reddy wrote:
> Thanks,
> In OS X following works:
> subprocess.call(['pip3','install','pyxform'])
>
> Does same works  works for Ubuntu/linux also?
>  
> for windows, I am assuming 'python' refers to python 3 inside QGIS3.

Yep:

import subprocess
subprocess.call(['pip3','install','pyxform','--user'])

works too.

But as said, not sure if all distro's call the python3 pip: pip3 (and/or
python3). I still would be afraid to force a specific version of a
module, as I would be afraid to override some excisting one.
I do like your solution though, the libs end up in my user site-packages
dir:
/home/richard/.local/lib/python3.5/site-packages


Ah: http://zetcode.com/lang/python/introspection/
>>> import sys
>>> sys.executable
'/usr/bin/python3'

gives you your actual python exe...

I wonder if you can create an venv from within QGIS in the subprocess way.
Mmm, this works (from within QGIS):

>>> import sys
>>> py_exe = sys.executable
>>> venv_path =
'/home/richard/.local/share/QGIS/QGIS3/profiles/default/venv'
>>> subprocess.call([py_exe,'-m','venv', venv_path])

Had to install python3-venv though:
sudo apt-get install python3-venv

sourcing it does not work with me:

>>> subprocess.call(['source', venv_path+'/bin/activate'])
FileNotFoundError: [Errno 2] No such file or directory: 'source'

Now somebody with more knowledge of the different python-path/lib-dir's
should come in, to tell us if this is usable after activating that venv...

Not sure if this would work with the already running python interpreter
in QGIS...

Idea's?

Regards,

Richard Duivenvoorde
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

axel_ande
Hi, 

I added this to my code just 3 days ago :)

import sys

try:
    import matplotlib
except:
    import subprocess
    print('installing matplotlib')
    subprocess.call([sys.exec_prefix + '/python', "-m", 'pip', 'install', 'matplotlib'])
    import matplotlib
    print('installation completed')

I dont know if it is a better or worse solution, I've only tested it on a Windows machine with the standalone version and the osgeo4w installation.

I tried calling sys.executable first, instead of sys.exec_prefix + '/python', which resulted in a new qgis window started :)

Axel

On 7 July 2018 at 14:23, Richard Duivenvoorde <[hidden email]> wrote:
On 07/07/2018 01:26 PM, shiva reddy wrote:
> Thanks,
> In OS X following works:
> subprocess.call(['pip3','install','pyxform'])
>
> Does same works  works for Ubuntu/linux also?

> for windows, I am assuming 'python' refers to python 3 inside QGIS3.

Yep:

import subprocess
subprocess.call(['pip3','install','pyxform','--user'])

works too.

But as said, not sure if all distro's call the python3 pip: pip3 (and/or
python3). I still would be afraid to force a specific version of a
module, as I would be afraid to override some excisting one.
I do like your solution though, the libs end up in my user site-packages
dir:
/home/richard/.local/lib/python3.5/site-packages


Ah: http://zetcode.com/lang/python/introspection/
>>> import sys
>>> sys.executable
'/usr/bin/python3'

gives you your actual python exe...

I wonder if you can create an venv from within QGIS in the subprocess way.
Mmm, this works (from within QGIS):

>>> import sys
>>> py_exe = sys.executable
>>> venv_path =
'/home/richard/.local/share/QGIS/QGIS3/profiles/default/venv'
>>> subprocess.call([py_exe,'-m','venv', venv_path])

Had to install python3-venv though:
sudo apt-get install python3-venv

sourcing it does not work with me:

>>> subprocess.call(['source', venv_path+'/bin/activate'])
FileNotFoundError: [Errno 2] No such file or directory: 'source'

Now somebody with more knowledge of the different python-path/lib-dir's
should come in, to tell us if this is usable after activating that venv...

Not sure if this would work with the already running python interpreter
in QGIS...

Idea's?

Regards,

Richard Duivenvoorde
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

shiva reddy
Thanks,
But it does not work in MacOS


Thanks & Regards
Shiva Reddy K.
Scientist/Engineer 'SC'
Indian Institute of Remote Sensing,
Indian Space Research Organisation
Department of Space
4-Kalidas Road
Dehradun
mobile: 8791806093


On Sun, Jul 8, 2018 at 11:12 AM Axel Andersson <[hidden email]> wrote:
Hi, 

I added this to my code just 3 days ago :)

import sys

try:
    import matplotlib
except:
    import subprocess
    print('installing matplotlib')
    subprocess.call([sys.exec_prefix + '/python', "-m", 'pip', 'install', 'matplotlib'])
    import matplotlib
    print('installation completed')

I dont know if it is a better or worse solution, I've only tested it on a Windows machine with the standalone version and the osgeo4w installation.

I tried calling sys.executable first, instead of sys.exec_prefix + '/python', which resulted in a new qgis window started :)

Axel

On 7 July 2018 at 14:23, Richard Duivenvoorde <[hidden email]> wrote:
On 07/07/2018 01:26 PM, shiva reddy wrote:
> Thanks,
> In OS X following works:
> subprocess.call(['pip3','install','pyxform'])
>
> Does same works  works for Ubuntu/linux also?

> for windows, I am assuming 'python' refers to python 3 inside QGIS3.

Yep:

import subprocess
subprocess.call(['pip3','install','pyxform','--user'])

works too.

But as said, not sure if all distro's call the python3 pip: pip3 (and/or
python3). I still would be afraid to force a specific version of a
module, as I would be afraid to override some excisting one.
I do like your solution though, the libs end up in my user site-packages
dir:
/home/richard/.local/lib/python3.5/site-packages


Ah: http://zetcode.com/lang/python/introspection/
>>> import sys
>>> sys.executable
'/usr/bin/python3'

gives you your actual python exe...

I wonder if you can create an venv from within QGIS in the subprocess way.
Mmm, this works (from within QGIS):

>>> import sys
>>> py_exe = sys.executable
>>> venv_path =
'/home/richard/.local/share/QGIS/QGIS3/profiles/default/venv'
>>> subprocess.call([py_exe,'-m','venv', venv_path])

Had to install python3-venv though:
sudo apt-get install python3-venv

sourcing it does not work with me:

>>> subprocess.call(['source', venv_path+'/bin/activate'])
FileNotFoundError: [Errno 2] No such file or directory: 'source'

Now somebody with more knowledge of the different python-path/lib-dir's
should come in, to tell us if this is usable after activating that venv...

Not sure if this would work with the already running python interpreter
in QGIS...

Idea's?

Regards,

Richard Duivenvoorde
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer


_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

Luke
In reply to this post by axel_ande
Rather than using a subprocess and needing to figure out which pip/python etc., have you tried importing and using the pip module directly? Something like:

import sys

try:
    import matplotlib
except:
    import pip
    pip.main(['install', 'matplotlib'])
    import matplotlib
Luke

On Sun., 8 Jul. 2018, 15:42 Axel Andersson, <[hidden email]> wrote:
Hi, 

I added this to my code just 3 days ago :)

import sys

try:
    import matplotlib
except:
    import subprocess
    print('installing matplotlib')
    subprocess.call([sys.exec_prefix + '/python', "-m", 'pip', 'install', 'matplotlib'])
    import matplotlib
    print('installation completed')

I dont know if it is a better or worse solution, I've only tested it on a Windows machine with the standalone version and the osgeo4w installation.

I tried calling sys.executable first, instead of sys.exec_prefix + '/python', which resulted in a new qgis window started :)

Axel

On 7 July 2018 at 14:23, Richard Duivenvoorde <[hidden email]> wrote:
On 07/07/2018 01:26 PM, shiva reddy wrote:
> Thanks,
> In OS X following works:
> subprocess.call(['pip3','install','pyxform'])
>
> Does same works  works for Ubuntu/linux also?

> for windows, I am assuming 'python' refers to python 3 inside QGIS3.

Yep:

import subprocess
subprocess.call(['pip3','install','pyxform','--user'])

works too.

But as said, not sure if all distro's call the python3 pip: pip3 (and/or
python3). I still would be afraid to force a specific version of a
module, as I would be afraid to override some excisting one.
I do like your solution though, the libs end up in my user site-packages
dir:
/home/richard/.local/lib/python3.5/site-packages


Ah: http://zetcode.com/lang/python/introspection/
>>> import sys
>>> sys.executable
'/usr/bin/python3'

gives you your actual python exe...

I wonder if you can create an venv from within QGIS in the subprocess way.
Mmm, this works (from within QGIS):

>>> import sys
>>> py_exe = sys.executable
>>> venv_path =
'/home/richard/.local/share/QGIS/QGIS3/profiles/default/venv'
>>> subprocess.call([py_exe,'-m','venv', venv_path])

Had to install python3-venv though:
sudo apt-get install python3-venv

sourcing it does not work with me:

>>> subprocess.call(['source', venv_path+'/bin/activate'])
FileNotFoundError: [Errno 2] No such file or directory: 'source'

Now somebody with more knowledge of the different python-path/lib-dir's
should come in, to tell us if this is usable after activating that venv...

Not sure if this would work with the already running python interpreter
in QGIS...

Idea's?

Regards,

Richard Duivenvoorde
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

shiva reddy
It does not work in python console of QGIS.
pip have __main__() methot that too for internal use.


Thanks & Regards
Shiva Reddy K.
Scientist/Engineer 'SD'
Indian Institute of Remote Sensing,
Indian Space Research Organisation
Department of Space
4-Kalidas Road
Dehradun
mobile: 8791806093


On Sun, Jul 8, 2018 at 11:58 AM Luke Pinner <[hidden email]> wrote:
Rather than using a subprocess and needing to figure out which pip/python etc., have you tried importing and using the pip module directly? Something like:

import sys

try:
    import matplotlib
except:
    import pip
    pip.main(['install', 'matplotlib'])
    import matplotlib
Luke

On Sun., 8 Jul. 2018, 15:42 Axel Andersson, <[hidden email]> wrote:
Hi, 

I added this to my code just 3 days ago :)

import sys

try:
    import matplotlib
except:
    import subprocess
    print('installing matplotlib')
    subprocess.call([sys.exec_prefix + '/python', "-m", 'pip', 'install', 'matplotlib'])
    import matplotlib
    print('installation completed')

I dont know if it is a better or worse solution, I've only tested it on a Windows machine with the standalone version and the osgeo4w installation.

I tried calling sys.executable first, instead of sys.exec_prefix + '/python', which resulted in a new qgis window started :)

Axel

On 7 July 2018 at 14:23, Richard Duivenvoorde <[hidden email]> wrote:
On 07/07/2018 01:26 PM, shiva reddy wrote:
> Thanks,
> In OS X following works:
> subprocess.call(['pip3','install','pyxform'])
>
> Does same works  works for Ubuntu/linux also?

> for windows, I am assuming 'python' refers to python 3 inside QGIS3.

Yep:

import subprocess
subprocess.call(['pip3','install','pyxform','--user'])

works too.

But as said, not sure if all distro's call the python3 pip: pip3 (and/or
python3). I still would be afraid to force a specific version of a
module, as I would be afraid to override some excisting one.
I do like your solution though, the libs end up in my user site-packages
dir:
/home/richard/.local/lib/python3.5/site-packages


Ah: http://zetcode.com/lang/python/introspection/
>>> import sys
>>> sys.executable
'/usr/bin/python3'

gives you your actual python exe...

I wonder if you can create an venv from within QGIS in the subprocess way.
Mmm, this works (from within QGIS):

>>> import sys
>>> py_exe = sys.executable
>>> venv_path =
'/home/richard/.local/share/QGIS/QGIS3/profiles/default/venv'
>>> subprocess.call([py_exe,'-m','venv', venv_path])

Had to install python3-venv though:
sudo apt-get install python3-venv

sourcing it does not work with me:

>>> subprocess.call(['source', venv_path+'/bin/activate'])
FileNotFoundError: [Errno 2] No such file or directory: 'source'

Now somebody with more knowledge of the different python-path/lib-dir's
should come in, to tell us if this is usable after activating that venv...

Not sure if this would work with the already running python interpreter
in QGIS...

Idea's?

Regards,

Richard Duivenvoorde
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

Luke
In reply to this post by Luke
My mistake. Importing pip is not supported and this functionality has been removed in pip>=10




On Sun., 8 Jul. 2018, 16:28 Luke Pinner, <[hidden email]> wrote:
Rather than using a subprocess and needing to figure out which pip/python etc., have you tried importing and using the pip module directly? Something like:

import sys

try:
    import matplotlib
except:
    import pip
    pip.main(['install', 'matplotlib'])
    import matplotlib
Luke

On Sun., 8 Jul. 2018, 15:42 Axel Andersson, <[hidden email]> wrote:
Hi, 

I added this to my code just 3 days ago :)

import sys

try:
    import matplotlib
except:
    import subprocess
    print('installing matplotlib')
    subprocess.call([sys.exec_prefix + '/python', "-m", 'pip', 'install', 'matplotlib'])
    import matplotlib
    print('installation completed')

I dont know if it is a better or worse solution, I've only tested it on a Windows machine with the standalone version and the osgeo4w installation.

I tried calling sys.executable first, instead of sys.exec_prefix + '/python', which resulted in a new qgis window started :)

Axel

On 7 July 2018 at 14:23, Richard Duivenvoorde <[hidden email]> wrote:
On 07/07/2018 01:26 PM, shiva reddy wrote:
> Thanks,
> In OS X following works:
> subprocess.call(['pip3','install','pyxform'])
>
> Does same works  works for Ubuntu/linux also?

> for windows, I am assuming 'python' refers to python 3 inside QGIS3.

Yep:

import subprocess
subprocess.call(['pip3','install','pyxform','--user'])

works too.

But as said, not sure if all distro's call the python3 pip: pip3 (and/or
python3). I still would be afraid to force a specific version of a
module, as I would be afraid to override some excisting one.
I do like your solution though, the libs end up in my user site-packages
dir:
/home/richard/.local/lib/python3.5/site-packages


Ah: http://zetcode.com/lang/python/introspection/
>>> import sys
>>> sys.executable
'/usr/bin/python3'

gives you your actual python exe...

I wonder if you can create an venv from within QGIS in the subprocess way.
Mmm, this works (from within QGIS):

>>> import sys
>>> py_exe = sys.executable
>>> venv_path =
'/home/richard/.local/share/QGIS/QGIS3/profiles/default/venv'
>>> subprocess.call([py_exe,'-m','venv', venv_path])

Had to install python3-venv though:
sudo apt-get install python3-venv

sourcing it does not work with me:

>>> subprocess.call(['source', venv_path+'/bin/activate'])
FileNotFoundError: [Errno 2] No such file or directory: 'source'

Now somebody with more knowledge of the different python-path/lib-dir's
should come in, to tell us if this is usable after activating that venv...

Not sure if this would work with the already running python interpreter
in QGIS...

Idea's?

Regards,

Richard Duivenvoorde
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

shiva reddy
Hi,
Finally I found following solution, which works in both windows and mac:


import subprocess
try:
        subprocess.call(['python3', '-m', 'pip', 'install','pyxform'])
        from pyxform.builder import create_survey_element_from_dict
        print('package is installed after python3')
    except:
        subprocess.call(['python3', '-m', 'pip', 'install','pyxform','--user'])
        print ("after python3 --user call")
        try:
            from pyxform.builder import create_survey_element_from_dict
        except:
            print('not able to install pyxform, install mannually')
  • First 'subprocess.call' works in Mac without permission issue.
  • Second 'subprocess.call' works in windows without permission issue.
Can anybody tries this code in ubuntu/linux and verify.


Thanks


If somebody can test this in other platform, I would be very glad.
Thanks & Regards
Shiva Reddy K.
Scientist/Engineer 'SD'
Indian Institute of Remote Sensing,
Indian Space Research Organisation
Department of Space
4-Kalidas Road
Dehradun
mobile: 8791806093


On Sun, Jul 8, 2018 at 4:20 PM Luke Pinner <[hidden email]> wrote:
My mistake. Importing pip is not supported and this functionality has been removed in pip>=10




On Sun., 8 Jul. 2018, 16:28 Luke Pinner, <[hidden email]> wrote:
Rather than using a subprocess and needing to figure out which pip/python etc., have you tried importing and using the pip module directly? Something like:

import sys

try:
    import matplotlib
except:
    import pip
    pip.main(['install', 'matplotlib'])
    import matplotlib
Luke

On Sun., 8 Jul. 2018, 15:42 Axel Andersson, <[hidden email]> wrote:
Hi, 

I added this to my code just 3 days ago :)

import sys

try:
    import matplotlib
except:
    import subprocess
    print('installing matplotlib')
    subprocess.call([sys.exec_prefix + '/python', "-m", 'pip', 'install', 'matplotlib'])
    import matplotlib
    print('installation completed')

I dont know if it is a better or worse solution, I've only tested it on a Windows machine with the standalone version and the osgeo4w installation.

I tried calling sys.executable first, instead of sys.exec_prefix + '/python', which resulted in a new qgis window started :)

Axel

On 7 July 2018 at 14:23, Richard Duivenvoorde <[hidden email]> wrote:
On 07/07/2018 01:26 PM, shiva reddy wrote:
> Thanks,
> In OS X following works:
> subprocess.call(['pip3','install','pyxform'])
>
> Does same works  works for Ubuntu/linux also?

> for windows, I am assuming 'python' refers to python 3 inside QGIS3.

Yep:

import subprocess
subprocess.call(['pip3','install','pyxform','--user'])

works too.

But as said, not sure if all distro's call the python3 pip: pip3 (and/or
python3). I still would be afraid to force a specific version of a
module, as I would be afraid to override some excisting one.
I do like your solution though, the libs end up in my user site-packages
dir:
/home/richard/.local/lib/python3.5/site-packages


Ah: http://zetcode.com/lang/python/introspection/
>>> import sys
>>> sys.executable
'/usr/bin/python3'

gives you your actual python exe...

I wonder if you can create an venv from within QGIS in the subprocess way.
Mmm, this works (from within QGIS):

>>> import sys
>>> py_exe = sys.executable
>>> venv_path =
'/home/richard/.local/share/QGIS/QGIS3/profiles/default/venv'
>>> subprocess.call([py_exe,'-m','venv', venv_path])

Had to install python3-venv though:
sudo apt-get install python3-venv

sourcing it does not work with me:

>>> subprocess.call(['source', venv_path+'/bin/activate'])
FileNotFoundError: [Errno 2] No such file or directory: 'source'

Now somebody with more knowledge of the different python-path/lib-dir's
should come in, to tell us if this is usable after activating that venv...

Not sure if this would work with the already running python interpreter
in QGIS...

Idea's?

Regards,

Richard Duivenvoorde
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

Nyall Dawson
In reply to this post by shiva reddy
On Sat, 7 Jul 2018 at 16:10, shiva reddy <[hidden email]> wrote:

>
> Hello,
> I tried to install automatically during my pluign (QRealTime) installation by calling:
>
>  subprocess.call(['python','-m','install','pyxform','--user'])
>
> It worked many times in windows and it fails sometime.
> It don't work in OS X and I have not tested in ubuntu.
>
> What is the best practice for such situation?


Just taking a step back here -- is this something we actually want to
support/allow in plugins?

Seems to me like it opens the door for all sorts of security issues.

Nyall
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

shiva reddy
I don't think so. Since all plugins goes through approval mechanism, I think  it is not unsafe. Rather increase the plugin user base.
Even if we want to not allow this .In that case, We should have mechanism by which QGIS itself does installation of external dependencies based on approved plugin's metadata.
 It will ease the life of plugin developers for sure.

Shiva



Thanks & Regards
Shiva Reddy K.
Scientist/Engineer 'SD'
Indian Institute of Remote Sensing,
Indian Space Research Organisation
Department of Space
4-Kalidas Road
Dehradun
mobile: 8791806093


On Tue, Jul 10, 2018 at 4:17 AM Nyall Dawson <[hidden email]> wrote:
On Sat, 7 Jul 2018 at 16:10, shiva reddy <[hidden email]> wrote:
>
> Hello,
> I tried to install automatically during my pluign (QRealTime) installation by calling:
>
>  subprocess.call(['python','-m','install','pyxform','--user'])
>
> It worked many times in windows and it fails sometime.
> It don't work in OS X and I have not tested in ubuntu.
>
> What is the best practice for such situation?


Just taking a step back here -- is this something we actually want to
support/allow in plugins?

Seems to me like it opens the door for all sorts of security issues.

Nyall

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

Richard Duivenvoorde
On 07/10/2018 06:15 AM, shiva reddy wrote:
> I don't think so. Since all plugins goes through approval mechanism, I
> think  it is not unsafe. Rather increase the plugin user base.
> Even if we want to not allow this .In that case, We should have
> mechanism by which QGIS itself does installation of external
> dependencies based on approved plugin's metadata.
>  It will ease the life of plugin developers for sure.
>
> Shiva

>     Just taking a step back here -- is this something we actually want to
>     support/allow in plugins?
>
>     Seems to me like it opens the door for all sorts of security issues.
>
>     Nyall

@shiva: the 'approval mechanism' is done by humans, and a lot of work,
as there are a lot of dev releasing small changes of their plugins...
So I would not count on that for security.
We have been talking about running some (security) rules over the plugin
sources before approval, but... nobody implemented it yet.

@nyall: as I said earlier in this thread, we have been talking about
adding the pip-command in the metadata.txt. That is better, yes?

I agree with you, Shiva's way has some security issues, but I was
thinking that we could do a grep on the sources of a plugin
automatically, to search for the subprocess line, and check if something
else then pip is called?
It IS a friendly way of installing (well, as we keep it in user space...).
If we could make it being installed in a QGIS Virtual Env or even in a
venv per plugin that would be safer?

Others?

Regards,

Richard Duivenvoorde

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: External python package dependency in plugins

Tim Sutton-6
Hi



On 10 Jul 2018, at 08:03, Richard Duivenvoorde <[hidden email]> wrote:

On 07/10/2018 06:15 AM, shiva reddy wrote:
I don't think so. Since all plugins goes through approval mechanism, I
think  it is not unsafe. Rather increase the plugin user base.
Even if we want to not allow this .In that case, We should have
mechanism by which QGIS itself does installation of external
dependencies based on approved plugin's metadata.
 It will ease the life of plugin developers for sure.

Shiva

   Just taking a step back here -- is this something we actually want to
   support/allow in plugins?

   Seems to me like it opens the door for all sorts of security issues.

   Nyall

@shiva: the 'approval mechanism' is done by humans, and a lot of work,
as there are a lot of dev releasing small changes of their plugins...
So I would not count on that for security.
We have been talking about running some (security) rules over the plugin
sources before approval, but... nobody implemented it yet.

And when we do I think we will probably be chasing our tails a lot….


@nyall: as I said earlier in this thread, we have been talking about
adding the pip-command in the metadata.txt. That is better, yes?

Doesn’t that just move the issue to a different place. The fact is you can put anything you like in PyPi including some nefarious module that does rm -rf on your file system, then just pull it in from the metadata.txt requirements list and run it. So I dont think we gain anything.



I agree with you, Shiva's way has some security issues, but I was
thinking that we could do a grep on the sources of a plugin
automatically, to search for the subprocess line, and check if something
else then pip is called?
It IS a friendly way of installing (well, as we keep it in user space...).
If we could make it being installed in a QGIS Virtual Env or even in a
venv per plugin that would be safer?


There are many other valid uses of using subprocess (e.g. running some gdal command) so I don’t think that gains us much either.

The venv thing we discussed in the past there are issues (Martin can fill in blanks here too as he was part of the discussion):

* For a true ’sandbox for plugins’ solution, the QGIS python interpreter and the one used by the plugin need to be the same otherwise you cannot use the API.

* Plugins should be able to use code from each other and build on each other (e.g. processing plugins) so they should be able to import each other’s modules. Or maybe we should make a rule that we dont support this.

* Env’s should be per-plugin so that two plugins could have different listed dependency versions which goes against the above point

I think the whole thing is too messy other than perhaps having one global (or one per profile) based venv with has system modules included from QGIS python lib.

My preferred solution is till to in time to refine the plugins to ‘official’ and ‘unofficial’ plugins. Official plugins would undergo some formal (paid? Manual?) detailed code review and are accepted as safe. Apple does this in their walled garden (no doubt with lots of automation) and though many FOSS people complain about walled gardens, there are valid, good reasons for their existence. We could also provide the ‘Wild West’ garden for unofficial / unreviewed plugins so that nobody loses out…

Regards

Tim



Others?

Regards,

Richard Duivenvoorde

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer








Tim Sutton

Co-founder: Kartoza
Ex Project chair: QGIS.org

Visit http://kartoza.com to find out about open source:

Desktop GIS programming services
Geospatial web development
GIS Training
Consulting Services

Skype: timlinux 
IRC: timlinux on #qgis at freenode.net


_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

signature.asc (499 bytes) Download Attachment