Sublime Forum

ST3: Loading plugins from "Installed Packages" may fail

#1

Sorry, the subject input is not long enough to describe the full issue.

Anyway, loading plugins from “Installed Packages” archives fails when a directory with the same name exists in “Packages”.

Steps to reproduce:

  1. Create a “TestPackage.sublime-package” zip archive with a Python plugin file named “test.py” with the following content:
print("testing")
  1. Place this package into your “Installed Packages” dir and restart ST. You will find the following in the console:

reloading plugin TestPackage.test testing

  1. Now, go to your “Packages” dir and create a folder named “TestPackage”. When restarting ST you will find the following:

reloading plugin TestPackage.test Traceback (most recent call last): File "C:\Program Files\Sublime Text 3 Portable Beta\sublime_plugin.py", line 73, in reload_plugin m = importlib.import_module(modulename) File "X/importlib/__init__.py", line 88, in import_module File "<frozen importlib._bootstrap>", line 1577, in _gcd_import File "<frozen importlib._bootstrap>", line 1558, in _find_and_load File "<frozen importlib._bootstrap>", line 1522, in _find_and_load_unlocked ImportError: No module named 'TestPackage.test'

Expected behaviour:
“test.py” loads successfully and prints “testing”. I know that the Packages dir is supposed to override the Installed Packages but it should only override those (files) if they actually exist.

Tested on Windows 7x64 with ST3 3035 Portable.

On issue tracker: github.com/SublimeText/Issues/issues/55

0 Likes

Dev Build 3038
#2

As of 3043 the particular error in this issue has been fixed (loading modules when a folder with an installed package’s name exists fails) but overriding installed packages is still not working that great.

For instance (assuming the same file structure as above):

  • Create a file named Packages/TestPackage/test.py and put something like print("not testing") in it. Console after restarting/saving:

reloading plugin TestPackage.test testing

  • Create a file named Packages/TestPackage/test2.py and enter something like print("testing again") (contents don’t matter). Console after restarting/saving:

[code]reloading plugin TestPackage.test2
Traceback (most recent call last):
File “C:\Program Files\Sublime Text 3 Portable Beta\sublime_plugin.py”, line 73, in reload_plugin
m = importlib.import_module(modulename)
File “X/importlib/init.py”, line 88, in import_module
File “”, line 1577, in _gcd_import
File “”, line 1558, in _find_and_load
File “”, line 1522, in _find_and_load_unlocked
ImportError: No module named ‘TestPackage.test2’

[/code]


I actually wanted to test this but due to the above error it wouldn’t work: Can I import moduels from installed packages from the same “uninstalled” package? E.g., would from . import test work in test2.py?

0 Likes

#3

FichteFoll: I believe these issues should be fixed in 3044, please let me know if they aren’t

0 Likes

#4

I’m still experiencing exactly the same issues as above with 3044:

reloading plugin TestPackage.test testing ... other packages ...] reloading plugin TestPackage.test2 Traceback (most recent call last): File "C:\Program Files\Sublime Text 3 Portable Beta\sublime_plugin.py", line 73, in reload_plugin m = importlib.import_module(modulename) File "X/importlib/__init__.py", line 88, in import_module File "<frozen importlib._bootstrap>", line 1577, in _gcd_import File "<frozen importlib._bootstrap>", line 1558, in _find_and_load File "<frozen importlib._bootstrap>", line 1522, in _find_and_load_unlocked ImportError: No module named 'TestPackage.test2'

0 Likes

#5

Unfortunately, it’s working for me. My setup is:

test.py in Installed Packages/TestPackage.sublime-package
test.py in Packages/TestPackage
test2.py in Packages/TestPackage

Both test.py and test2.py are getting imported as expected here in 3044.

0 Likes

#6

On Linux it’s working as expected. Tested on LMDE.
On Windows, it doesn’t work at all - test.py seems to get imported without error but does not override, test2.py fails to import - as FichteFoll reported. Tested on Win8 x64 and WinXP 32-bit, with portable and installed versions of ST 3044.

0 Likes

#7

Thanks for the reports FichteFoll and Drifter, 3045 is out now fixing the issue on Windows

0 Likes

#8

Yes, confirmed to be working now with 3045. And even more, this is actually possible:

test.py in Installed Packages/TestPackage.sublime-package:

print("testing") some_var = "something"

test.py in Packages/TestPackage:

[code]print(“before import”)

from . import test

print("this module: " + file)
print("installed module: " + test.file)

print("some_var before: " + test.some_var)
test.some_var = “nothing”
print(“overridden”)
[/code]

Which gives me this in the console:

reloading plugin TestPackage.test before import testing this module: C:\Program Files\Sublime Text 3 Portable Beta\Data\Installed Packages\TestPackage.sublime-package/TestPackage.test installed module: C:\Program Files\Sublime Text 3 Portable Beta\Data\Installed Packages\TestPackage.sublime-package\test.py some_var before: something overridden

And in the console I can do this:

[code]>>> import TestPackage.test

dir(TestPackage.test)
builtins’, ‘doc’, ‘file’, ‘loader’, ‘name’, ‘package’, ‘path’, ‘test’]

TestPackage.test.test
<module ‘TestPackage.test.test’ from ‘C:\Program Files\Sublime Text 3 Portable Beta\Data\Installed Packages\TestPackage.sublime-package\test.py’>

TestPackage.test.test.some_var
‘nothing’[/code]

Yes, cool stuff because now I can for example override only parts of a single module/plugin and just import the rest.

However, it seems that this introduced a regression in terms of relative imports as seen here: github.com/wbond/sublime_packag … issues/490 (Windows 8)
In particular:

reloading plugin Package Control.Package Control Traceback (most recent call last): File "C:\Users\Manuel\Programs\Sublime Text 3\sublime_plugin.py", line 73, in reload_plugin m = importlib.import_module(modulename) File "X/importlib/__init__.py", line 88, in import_module File "<frozen importlib._bootstrap>", line 1577, in _gcd_import File "<frozen importlib._bootstrap>", line 1558, in _find_and_load File "<frozen importlib._bootstrap>", line 1525, in _find_and_load_unlocked File "C:\Users\Manuel\Programs\Sublime Text 3\sublime_plugin.py", line 663, in load_module exec(compile(source, source_path, 'exec'), mod.__dict__) File "C:\Users\Manuel\Programs\Sublime Text 3\Data\Packages\Package Control\Package Control.py", line 60, in <module> from .package_control import reloader ImportError: No module named 'Package Control.Package Control.package_control' plugins loaded

Note that this works fine on my machine.

0 Likes

#9

This should now be fixed in 3046.

The observed issue was caused by having a Package Control.sublime-package file, as well as Package Control extracted. The importer in sublime_plugin.py wasn’t correctly distinguishing between packages and modules, which caused relative imports to work incorrectly.

0 Likes

#10

With 3049 (and I think 3046 because that’s probably the version I had before) the outlined behavior in my previous post no longer works. It now prints:

reloading plugin TestPackage.test before import Traceback (most recent call last): File "C:\Program Files\Sublime Text 3 Portable Beta\sublime_plugin.py", line 73, in reload_plugin m = importlib.import_module(modulename) File "X/importlib/__init__.py", line 88, in import_module File "<frozen importlib._bootstrap>", line 1577, in _gcd_import File "<frozen importlib._bootstrap>", line 1558, in _find_and_load File "<frozen importlib._bootstrap>", line 1525, in _find_and_load_unlocked File "C:\Program Files\Sublime Text 3 Portable Beta\sublime_plugin.py", line 671, in load_module exec(compile(source, source_path, 'exec'), mod.__dict__) File "C:\Program Files\Sublime Text 3 Portable Beta\Data\Packages\TestPackage\test.py", line 3, in <module> from . import test ImportError: cannot import name test

This is essentially crutial because relative imports don’t seem to work at all right now, not even with another module name.

I need to investigate this further though because it seems to work for Package Control. :S

0 Likes

#11

I decided to take back on this since I could not get relative importing to work on my packages when updating to ST3.

This post contains a lot of (necessary or not) information for reproduction, including many tracebacks. Please take a look at the critical example 3: ST swallows runtime exceptions from imported files when importing, so you never actually know where the error is!


Everything that is following has been run on Windows 7x64 with a fresh Sublime Text Portable x64 Build 3059.

Note that TestPackage.sublime-package (pretty much any file that fails to load) always seems to remain opened by ST because I have to close it to delete the file.

  1. Example 1

File Structure:

Data/
+ Packages/
| + TestPackage/
| | + subpackage/
| | | + test4.py
| | + testsub.py
+ Installed Packages/
| + TestPackage.sublime-package
| | + test.py[/code]

**test.py**
:
[code]print("loading test in sublime-package")

testsub.py:

[code]from .subpackage import test4

print(test4.variable)[/code]

test4.py:

[code]print(“we subtesting now”)

variable = “test4”
[/code]

Console: (when restarting)

reloading plugin TestPackage.test loading test in sublime-package reloading plugin TestPackage.testsub Traceback (most recent call last): File "C:\Program Files\Sublime Text 3 Portable Beta TEST\sublime_plugin.py", line 73, in reload_plugin m = importlib.import_module(modulename) File "X/importlib/__init__.py", line 88, in import_module File "<frozen importlib._bootstrap>", line 1577, in _gcd_import File "<frozen importlib._bootstrap>", line 1558, in _find_and_load File "<frozen importlib._bootstrap>", line 1525, in _find_and_load_unlocked File "C:\Program Files\Sublime Text 3 Portable Beta TEST\sublime_plugin.py", line 671, in load_module exec(compile(source, source_path, 'exec'), mod.__dict__) File "C:\Program Files\Sublime Text 3 Portable Beta TEST\Data\Packages\TestPackage\testsub.py", line 1, in <module> from .subpackage import test4 File "<frozen importlib._bootstrap>", line 1558, in _find_and_load File "<frozen importlib._bootstrap>", line 1525, in _find_and_load_unlocked File "C:\Program Files\Sublime Text 3 Portable Beta TEST\sublime_plugin.py", line 671, in load_module exec(compile(source, source_path, 'exec'), mod.__dict__) UnboundLocalError: local variable 'source' referenced before assignment

Expected behavior: No exception and “we subtesting now\ntest4” in the console.

\

  1. Example 2 (more advanced but worked on 3045)

File Structure:

Data/
+ Packages/
| + TestPackage/
| | + test.py
+ Installed Packages/
| + TestPackage.sublime-package
| | + test.py[/code]

**test.py**
 (in sublime-package):
[code]print("loading test in sublime-package")

variable = "test.sublime-package"

testsub.py (in Packages/TestPackage):

[code]print(“before import”)

from . import test

print(dir(test))

print("this module: " + file)
print("installed module: " + test.file)

print("variable before: " + test.variable)
test.variable = “changed!”
print("variable now: " + test.variable)
[/code]

Console: (when restarting)

reloading plugin TestPackage.test before import Traceback (most recent call last): File "C:\Program Files\Sublime Text 3 Portable Beta TEST\sublime_plugin.py", line 73, in reload_plugin m = importlib.import_module(modulename) File "X/importlib/__init__.py", line 88, in import_module File "<frozen importlib._bootstrap>", line 1577, in _gcd_import File "<frozen importlib._bootstrap>", line 1558, in _find_and_load File "<frozen importlib._bootstrap>", line 1525, in _find_and_load_unlocked File "C:\Program Files\Sublime Text 3 Portable Beta TEST\sublime_plugin.py", line 671, in load_module exec(compile(source, source_path, 'exec'), mod.__dict__) File "C:\Program Files\Sublime Text 3 Portable Beta TEST\Data\Packages\TestPackage\test.py", line 3, in <module> from . import test ImportError: cannot import name test

Expected behavior: Same as mentioned above: viewtopic.php?f=3&t=12564#p50283

\

  1. Example 3 (THIS IS CRITICAL)

File Structure:

Data/
Data/
+ Packages/
| + TestPackage/
| | + sub/
| | | + other_module.py
| | + testsub.py[/code]

**testsub.py**:
[code]from .sub import other_module

print("loading test in sublime-package")

variable = "test.sublime-package"

other_module.py:

[code]print(“importing other module”)

from doesntexist import doesntexisteither # ImportError!
variable = “testing~”

print “SyntaxError”

[/code]


Console: (when restarting)

reloading plugin TestPackage.testsub importing other module Traceback (most recent call last): File "C:\Program Files\Sublime Text 3 Portable Beta TEST\sublime_plugin.py", line 73, in reload_plugin m = importlib.import_module(modulename) File "X/importlib/__init__.py", line 88, in import_module File "<frozen importlib._bootstrap>", line 1577, in _gcd_import File "<frozen importlib._bootstrap>", line 1558, in _find_and_load File "<frozen importlib._bootstrap>", line 1525, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper File "<frozen importlib._bootstrap>", line 1023, in load_module File "<frozen importlib._bootstrap>", line 1004, in load_module File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper File "<frozen importlib._bootstrap>", line 869, in _load_module File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed File "C:\Program Files\Sublime Text 3 Portable Beta TEST\Data\Packages\TestPackage\testsub.py", line 1, in <module> from .sub import other_module ImportError: cannot import name other_module

Expected behavior: Show the actual exception from other_module: File "C:\Program Files\Sublime Text 3 Portable Beta TEST\Data\Packages\TestPackage\testsub.py", line 1, in <module> from .sub import other_module ImportError: cannot import name other_module

Note how the ImportError from within other_module is swallowed. It took me hours to try to debug why my import just would be found until I suspected the actual exception is swallowed. I eventually stopped debugging for half a yeah (basically since my last post in this thread) and discovered this only just now.


other_module.py (with SyntaxError):

[code]print(“importing other module”)

from doesntexist import doesntexisteither # ImportError!

variable = “testing~”

print “SyntaxError”
[/code]

The console looks like this:

reloading plugin TestPackage.testsub Traceback (most recent call last): File "C:\Program Files\Sublime Text 3 Portable Beta TEST\sublime_plugin.py", line 73, in reload_plugin m = importlib.import_module(modulename) File "X/importlib/__init__.py", line 88, in import_module File "<frozen importlib._bootstrap>", line 1577, in _gcd_import File "<frozen importlib._bootstrap>", line 1558, in _find_and_load File "<frozen importlib._bootstrap>", line 1525, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper File "<frozen importlib._bootstrap>", line 1023, in load_module File "<frozen importlib._bootstrap>", line 1004, in load_module File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper File "<frozen importlib._bootstrap>", line 869, in _load_module File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed File "C:\Program Files\Sublime Text 3 Portable Beta TEST\Data\Packages\TestPackage\testsub.py", line 1, in <module> from .sub import other_module File "C:\Program Files\Sublime Text 3 Portable Beta TEST\Data\Packages\TestPackage\sub\other_module.py", line 6 print "SyntaxError" ^ SyntaxError: invalid syntax

This is likely because Syntax errors are detected at parse-time, not at runtime. I have to wrap the entire imported file in a try-except in order to find this.


other_module.py wrapped in try-except:

[code]try:
print(“importing other module”)

from doesntexist import doesntexisteither # ImportError!
variable = "testing~"

# print "SyntaxError"

except:
import traceback
print("=" * 30)
traceback.print_exc()
print("=" * 30)
raise
[/code]

Console: (after restart)

[code]reloading plugin TestPackage.testsub
importing other module

Traceback (most recent call last):
File “C:\Program Files\Sublime Text 3 Portable Beta TEST\Data\Packages\TestPackage\sub\other_module.py”, line 4, in
from doesntexist import doesntexisteither # ImportError!
ImportError: No module named ‘doesntexist’

Traceback (most recent call last):
File “C:\Program Files\Sublime Text 3 Portable Beta TEST\sublime_plugin.py”, line 73, in reload_plugin
m = importlib.import_module(modulename)
File “X/importlib/init.py”, line 88, in import_module
File “”, line 1577, in _gcd_import
File “”, line 1558, in _find_and_load
File “”, line 1525, in _find_and_load_unlocked
File “”, line 586, in _check_name_wrapper
File “”, line 1023, in load_module
File “”, line 1004, in load_module
File “”, line 562, in module_for_loader_wrapper
File “”, line 869, in _load_module
File “”, line 313, in _call_with_frames_removed
File “C:\Program Files\Sublime Text 3 Portable Beta TEST\Data\Packages\TestPackage\testsub.py”, line 1, in
from .sub import other_module
ImportError: cannot import name other_module[/code]

0 Likes

#12

I have the same problem and I keep getting ImportErrors for .py files imported from the plugin folder

ImportError: No module named 'foo'
0 Likes

#13

[quote=“kblomqvist”]I have the same problem and I keep getting ImportErrors for .py files imported from the plugin folder

ImportError: No module named 'foo'

Since you are getting a different error message I suppose you are having a different issue. How can I reproduce your problem? Especially your project setup with files and what the import statement is.

0 Likes

#14

Example 3 has already been fixed in 3061 where the Python API was updated to 3.3 and fixed the upstream bug bugs.python.org/issue15111.

See also: github.com/SublimeText/Issues/i … t-38028215

0 Likes