Sublime Forum

CodeIntel ported from Open Komodo Editor

#74

@squ1b3r, is that traceback from editing a css file?

@alexrussell, for the compiler you’ll need Vusual Studio (probably it can be compiled with VS express version, but I’m not entirely sure). You’ll need python with it’s libraries and headers (same version and architecture as the one binded with Sulime Text 2 for windows) and you’ll also need to compile libpcre (as a static library) since SilverCity needs to link against it. Next, doing “python setup.py build” should compile everything after going out to the console VS provides for building stuff (32 and 64 bits, depending). Please check “build.sh” which contains the steps for building stuff in Mac OS X and Linux, but I dont think I’m missing anything… except the extensions for windows libraries will be “.pyd”, not “.so”. If you can buil using this procedure, please post about it here. Consider building both 32 and 64 bits libraries if you can.

For building under Linux, you’ll have to run build.sh, but the only catch is you’ll need to compile python yourself with UCS-4 (for 4 bytes Unicode strings)… also, in Linux, you’ll also have to preferably build libpcre yourself too (as a static library) to avoid having it as an external dependency.

Please, if anyone can build these for 32 and 64 bits architectures in these platforms, it would be very much appreciated.

0 Likes

#75

Hello @kronuz,

I tried to build it on a Ubuntu 11.04 machine:

(11:52:01) - douglas at shoryuken in src
$ uname -a
Linux shoryuken 2.6.38-10-generic-pae #46-Ubuntu SMP Tue Jun 28 16:54:49 UTC 2011 i686 i686 i386 GNU/Linux

(11:53:12) - douglas at shoryuken in src
$ cat /etc/issue
Ubuntu 11.04 \n \l

And im having these problems:

(11:51:43) - douglas at shoryuken in src
$ ./build.sh &> build_problems.log

Here are the link to a paste of the log ( in short, im having problems with LexTCL.cxx or something like that).

http://paste.archlinux-br.org/1784

I tried to look in the openkomodo code, but seems it is very different from SublimeCodeIntel (im talking about contrib/patches/scintilla).

Is there something else that i can do ?

Thanks !

0 Likes

#76

@dsarch, try again, but this time delete that LexTCL.cxx file. There should be a LexTcl.cxx (note lowercases) but not a LexTCL.cxx. I think I forgot to delete it in the build.sh (after copying the files from “src” for scintilla)

0 Likes

#77

No. I’ve got it on start of ST2

0 Likes

#78

Thanks @kronuz,

It worked =)

I added this line code after line 16 in build.sh:

find . -name "LexTCL*" | xargs rm

Then i just ./build.sh and it worked - the problem was the LexTCL.cxx file that comes from scintilla (which is unpacked everytime build.sh runs).

How do you want me to send the libs to you ?

Thanks again for the hard work !

0 Likes

#79

Hello @kronuz,

Also to make it work (i dont care for css support now), i had to comment these lines (17-21):

  9 class _CSSLexerTemplate(Lexer.Lexer):                                          
 10     def __init__(self, properties):                                            
 11         self._properties = properties                                          
 12         self._lexer = find_lexer_module_by_id(SCLEX_CSS)                       
 13         self._keyword_lists =                                                 
 14             WordList(Keywords.css_keywords),                                   
 15             WordList(Keywords.css_pseudo_classes),                             
 16             WordList(Keywords.css_keywords_2),                                 
 17             #WordList(Keywords.css_properties_3),                              
 18             #WordList(Keywords.css_pseudo_elements),                           
 19             #WordList(Keywords.css_browser_specific_properties),               
 20             #WordList(Keywords.css_browser_specific_pseudo_classes),           
 21             #WordList(Keywords.css_browser_specific_pseudo_elements),

In SublimeCodeIntel/libs/SilverCity/CSS.py.

Also, i just noticed i can skip css using this config:

"codeintel_disabled_languages": 'css']

Thanks !

0 Likes

#80

[quote=“Kronuz”]@squ1b3r, is that traceback from editing a css file?

@alexrussell, for the compiler you’ll need Vusual Studio (probably it can be compiled with VS express version, but I’m not entirely sure). You’ll need python with it’s libraries and headers (same version and architecture as the one binded with Sulime Text 2 for windows) and you’ll also need to compile libpcre (as a static library) since SilverCity needs to link against it. Next, doing “python setup.py build” should compile everything after going out to the console VS provides for building stuff (32 and 64 bits, depending). Please check “build.sh” which contains the steps for building stuff in Mac OS X and Linux, but I dont think I’m missing anything… except the extensions for windows libraries will be “.pyd”, not “.so”. If you can buil using this procedure, please post about it here. Consider building both 32 and 64 bits libraries if you can.

For building under Linux, you’ll have to run build.sh, but the only catch is you’ll need to compile python yourself with UCS-4 (for 4 bytes Unicode strings)… also, in Linux, you’ll also have to preferably build libpcre yourself too (as a static library) to avoid having it as an external dependency.

Please, if anyone can build these for 32 and 64 bits architectures in these platforms, it would be very much appreciated.[/quote]

Eep, I think that might end up being a bit too much for me to do. Was kinda hoping download and install gcc (or whatever), run build.bat or something! I’ll have a go at downloading VS etc.

0 Likes

#81

@dsarch, that is great! What’s the architecture you compiled it for? is it 32 or 64 bytes? Can you compile both architectures? …my email is german dot mb at gmail, please send it over. Thank you!

I have now noticed that other bug that suddenly started to happen for me too… but I have been really busy these days to dig into it. I’ll soon try to fix that issue, which is causing master’s head to fail, and I’ll report back. In the mean time, I made a git tag for SublimeCodeIntel repo, for when the old binaries were working for Windows and Linux (named it version 1.0). I’ll try to start maintaining a CHANGELOG file too.

0 Likes

#82

Brilliant news! This thing is amazing!

0 Likes

#83

Gives me this error -

Reloading plugin /Users/Pravin/Library/Application Support/Sublime Text 2/Packages/CodeIntel/python_codeintel.py Traceback (most recent call last): File "./sublime_plugin.py", line 57, in reload_plugin File "./python_codeintel.py", line 86, in <module> from codeintel2.common import * File "/Users/Pravin/Library/Application Support/Sublime Text 2/Packages/CodeIntel/libs/codeintel2/common.py", line 76, in <module> import SilverCity File "/Users/Pravin/Library/Application Support/Sublime Text 2/Packages/CodeIntel/libs/SilverCity/__init__.py", line 19, in <module> LanguageInfo.do_registration() File "/Users/Pravin/Library/Application Support/Sublime Text 2/Packages/CodeIntel/libs/SilverCity/LanguageInfo.py", line 116, in do_registration import CSS File "/Users/Pravin/Library/Application Support/Sublime Text 2/Packages/CodeIntel/libs/SilverCity/CSS.py", line 42, in <module> class CSSHTMLGenerator(HTMLGenerator.SimpleHTMLGenerator, CSSHandler): File "/Users/Pravin/Library/Application Support/Sublime Text 2/Packages/CodeIntel/libs/SilverCity/CSS.py", line 50, in CSSHTMLGenerator def generate_html(self, file, buffer, lexer = CSSLexer()): File "/Users/Pravin/Library/Application Support/Sublime Text 2/Packages/CodeIntel/libs/SilverCity/CSS.py", line 26, in __init__ _CSSLexerTemplate.__init__(self, properties) File "/Users/Pravin/Library/Application Support/Sublime Text 2/Packages/CodeIntel/libs/SilverCity/CSS.py", line 17, in __init__ WordList(Keywords.css_properties_3), AttributeError: 'module' object has no attribute 'css_properties_3'

0 Likes

#84

@pravim, i posted a fix in the thread (i guess it was in the page 9), it is working fine for me there.

0 Likes

#85

@kronuz,

Done =)

Sent to your email, if you prefer, i can send a pull request on github also.

EDIT: Just sent a pull request with the updated libs (confirmed working on a i386 linux machine, i did not test the x86_64 but i dont see where it could fail)

Thanks for all your work in SublimeCodeIntel.

0 Likes

#86

Hi,

Apologies if this has been asked before, but is it possible to reference packages in Python eggs in the pythonExtraPaths configuration?

I tried the following in .codeintel/config in a project:

{
    "Python": {
        "python": "/usr/local/bin/python2.6",
        "pythonExtraPaths": 
              "/Users/optilude/Development/Plone/Optilux/second-edition/src/optilux.policy",
              "/Users/optilude/Development/Plone/Optilux/second-edition/src/optilux.cinemacontent",
              "/Users/optilude/Development/Plone/Optilux/second-edition/src/optilux.theme",
              "/Users/optilude/.buildout/eggs/Zope2-2.13.8-py2.6.egg",
              "/Users/optilude/.buildout/eggs/zope.testing-3.9.6-py2.6.egg",
              "/Users/optilude/.buildout/eggs/zope.site-3.9.2-py2.6.egg",
              "/Users/optilude/.buildout/eggs/zope.publisher-3.12.6-py2.6.egg",
              "/Users/optilude/.buildout/eggs/zope.processlifetime-1.0-py2.6.egg",
              "/Users/optilude/.buildout/eggs/zope.interface-3.6.3-py2.6-macosx-10.6-x86_64.egg",
              "/Users/optilude/.buildout/eggs/zope.component-3.9.5-py2.6.egg",
              ...
            ]
    }
}

But the packages in those eggs are not picked up. Note that those are all unzipped eggs, and most of them contain namespace packages.

Cheers,
Martin

0 Likes

#87

Hi @kronuz,

+1 for @optilude request, it is great when working in Zope/Plone projects.

Also, great to see you here @optilude, im anxious for your Plone4 book (bought it already in packetpub) =)

All the best !

0 Likes

#88

Okay, I upgraded SublimeCodeIntel to fix “the issue”. Please pull from github :smile:

Thanks to dsarch we now have working binaries for linux again! …only Windows to go.

@dsarch, I merged your pull request, however recently a new file was changed in scintilla (LexPerl.cxx) so a new linux build will be eventually necessary. I hope it’s not much difficult to compile there… could you outline the missing steps? like how you built python with UCS-4 and other bits? (as to include it somewhere in the README)

Regarding the eggs, how would that work? I don’t work much with eggs or .pth files …please if anyone knows exactly how they should be supported, explain and I’ll try to add support for it :wink:

0 Likes

#89

Hello @kronuz,

Done, updated the libs and pushed to github again.

Thanks !

0 Likes

#90

[quote=“Kronuz”]
Regarding the eggs, how would that work? I don’t work much with eggs or .pth files …please if anyone knows exactly how they should be supported, explain and I’ll try to add support for it :wink:[/quote]

Let me try to show you with a buildout.

First, download the file at gist.github.com/1140588 and save it in a new directory as buildout.cfg.

Then go into this directory and run:

$ python bootstrap.py
$ bin/buildout

At this point, you’ll have an ‘eggs’ directory with some files in it and a ‘bin/mkrelease’. This will look a bit like this:

#!/usr/local/bin/python2.6

import sys
sys.path[0:0] = 
    '/Users/optilude/tmp/testbuild/eggs/jarn.mkrelease-3.1-py2.6.egg',
    '/Users/optilude/tmp/testbuild/eggs/lazy-1.0-py2.6.egg',
    '/Users/optilude/tmp/testbuild/eggs/setuptools_git-0.4.2-py2.6.egg',
    '/Users/optilude/tmp/testbuild/eggs/setuptools_hg-0.2.1-py2.6.egg',
    '/Users/optilude/tmp/testbuild/eggs/distribute-0.6.19-py2.6.egg',
    ]

import jarn.mkrelease.mkrelease

if __name__ == '__main__':
    jarn.mkrelease.mkrelease.main()

This is a wrapper script that mangles sys.path with the “working set” of eggs, inserting those eggs directly on sys.path, before executing the main function. If you look at the items it adds to the path, they are directories with Python packages and modules and an EGG-INFO directory with egg metadata.

Note that in some cases (e.g. eggs/jarn.mkrelease-3.1-py2.6.egg/jarn/init.py) these will use setuptools/distribute namespace packages (with some boilerplate in init.py). Assuming setuptools is on sys.path, it should deal with these as sys.path is searched when modules are imported.

Another approach to get some test eggs would be to use virtualenv. For example:

$ virtualenv eggtest
$ cd eggtest
$ bin/easy_install jarn.mkrelease

At this point, we have in lib/python2.6/site-packages an easy-install.pth that looks like this:

import sys; sys.__plen = len(sys.path)
./distribute-0.6.10-py2.6.egg
./pip-0.7.2-py2.6.egg
./jarn.mkrelease-3.1-py2.6.egg
./lazy-1.0-py2.6.egg
./setuptools_git-0.4.2-py2.6.egg
./setuptools_hg-0.2.1-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

These .egg files are the same as the ones found in the eggs/ directory in the buildout. In this case, we rely on the scanning of .pth files in site-packages to tell Python to add the *.egg directories to sys.path, but the end result is the same.

Now, I tried to use a .codeintel/config like this:

{
    "Python": {
        "python": "/usr/local/bin/python2.6",
        "pythonExtraPaths": 
            "/Users/optilude/tmp/testbuild/eggs/jarn.mkrelease-3.1-py2.6.egg",
            "/Users/optilude/tmp/testbuild/eggs/lazy-1.0-py2.6.egg",
            "/Users/optilude/tmp/testbuild/eggs/setuptools_git-0.4.2-py2.6.egg",
            "/Users/optilude/tmp/testbuild/eggs/setuptools_hg-0.2.1-py2.6.egg",
            "/Users/optilude/tmp/testbuild/eggs/distribute-0.6.19-py2.6.egg"
        ]
    }
}

Unfortunately, this didn’t work (e.g. I couldn’t get import completion from the jarn namespace except when actually in a module inside jarn.mkrelease).

Ultimately, what I’d like to do is to write a buildout recipe that spits out such a file when buildout is run. However, the above doesn’t work. I saw some explicit code referencing .egg files and skipping them, but even commenting this out, it didn’t work. I tried to understand how the path loading works in the codeintel package, but I failed. :wink:

Martin

0 Likes

#91

Actually… I think the manually constructed list of .egg paths works, but the problem is that it doesn’t support namespace packages.

I tried with a much bigger set of eggs (a full Plone 4.1 buildout with some development tools), and discovered:

  • Eggs not using namespace packages work (e.g. I can import from ‘feedparser’)

  • For eggs using namespace packages, the first one listed (e.g. zope.i18nmessageid) works, but subsequent ones (e.g. zope.site) does not

If you want to test something like this, try using this buildout.cfg:

[buildout]
extends =
    http://dist.plone.org/release/4.1/versions.cfg
parts = instance

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
eggs = Plone

The script with paths is now in bin/instance.

Cheers,
Martin

0 Likes

#92

Mmm… I’ve spent a couple of hours trying to understand where the imports are actually made and evaluated, so as to try to see how to make it respect namespace packages, but I can’t really understand it (and debugging plugins without pdb is a pain). If anyone can give some insight into where the imports and sys.path scanning are happening and how they’re used, that may be useful.

Martin

0 Likes

#93

[quote=“sublimator”]Not sure what your trying to do atm but I don’t think Sublime even ships with distutils letalone setuptools/distribute required for namespace packages to work.
[/quote]

Which Python interpreter does it use? I assumed it was /usr/bin/python, but maybe not?

I tried to make this work on OS X, but installing it was a pain (it doesn’t support 64-bit Python). I got it running, but then couldn’t figure out how to set a breakpoint that was actually triggered.

[quote]

Are you talking something specific to the CodeIntel plugin or … ?[/quote]

Let’s say I type “from zope.”. When I press “.” the CodeIntel plugin starts looking for completions, i.e. modules and packages inside the zope package. Right now, it seems to take the first thing it can find (usually “zope.i18n” on my system). Somehow, it needs to realise there are lots of other things inside “zope.”.

Looking at the blobs in ~/.codeintel/db, I’m pretty sure it actually finds all the relevant packages and modules when the ‘pythonExtraPaths’ list lists a bunch of .egg directories. The problem has to be in the way it does the search over the database.

Martin

0 Likes