Sublime Forum

Python Improved - a better Python language definition

#7

In the year(s) of using ST with Python I also experienced some things that the syntax file did not cover. Maybe some of it hasn’t been fixed yet by you (didn’t check myself):

  • next keyword (Py3)
  • __bool__ function (Py3)
  • 0b100101 numeric literal notation
  • When splitting a def function definition into multiple lines, comments are not recognized

You might want to consider monitoring github.com/SublimeText/Issues/i … state=open :wink:

0 Likes

#8

@FichteFoll:

  • next is included in “support.function.builtin.python” - it’s possible your current color scheme doesn’t highlight that scope properly.

  • I just added __bool__, so next time you update it should be there.

  • PythonImproved does include support for binary literals, both regular (‘0b01100101’) and long (‘0b01100101L’)

  • I also just added support for comments within function definitions:

def myfunc(self,            # gotta have self
           param1="value",  # values are cool
           param2=True,     # or False, whatever
           **kwargs):       # you never know

@skuroda:
I’ve put a list of new/modified scopes in the README. I think I got most of the major ones, but I did leave out some minor housekeeping, regex tuning, and whatnot.

As always, if something doesn’t look quite right, or you have any feedback at all, please let me know!

0 Likes

#9

Edit:

@MattDMo I decided to go ahead and give your language a try. I think I can probably help you with the Python 3’s function annotations. I like what you are doing with this.

0 Likes

#10

I see why you were struggling with those annotations. They are flexible enough to make things a pain. I’ll take a crack at it. It may be one of those things that will require multiple iterations over it before everything is 100% covered (or near 100% covered), but if a couple initial passes can bring it up cleanly greater than the current 0%, it will be an improvement.

0 Likes

#11

[pre=#2D2D2D]def greet(name: str, age: int) -> str:
print(‘Hello {0}, you are {1} years old’.format(name, age))

instead of the standard

def greet(name, age):
print(…)[/pre]

How does it look?

0 Likes

#12

These theme files always make my head hurt, but here is a more complicated example; comments and all:

[pre=#2D2D2D]def foo(
(x1, y1: expression), # Is this a comment?
(
x2: 3 + 2, # why yes, that is a comment!
y2: “some random string” # Yet another
) = (None, None)
) -> “yes this is the result annotation as a string”:
print(‘Hello {0}, you are {1} years old’.format(name, age))[/pre]

0 Likes

#13

That definitely looks like a start. Can you send me a pull request?

I just saw you fixed the <string> issue in the comments at the top - I meant to do that with my last release, but must have missed it. Thanks!

0 Likes

#14

I always convert my themes to json, and the syntax broke my plugin’s conversion, so I went ahead and patched it.

I will issue a pull request within the hour. I will document the new scopes for the annotation separators as well.

0 Likes

#15

Pull request made.

0 Likes

#16

Just stepping into this (and promoting): There is also the possibility to convert syntax definitions to YAML instead of JSON which is much more flexible, doesn’t require {} all over the place, does not require “” all over the place and does not require you to escape everything all over the place. Oh, and it supports multi-line strings and stuff. Here is an example: github.com/SublimeText/AAAPacka … tmLanguage

If you encounter any bugs (especially with the syntax highlighting, because YAML is a beast to highlight with regular expressions), please report them on the repo.

0 Likes

#17

One thing I would love to see is a simple plist to yaml and yaml to plist plugin (without all of the other AAAPackageDev stuff I don’t use). Maybe I will take a look and strip it out for personal use. I haven’t played with yaml much, but I think I would like to.

0 Likes

#18

The only relevant files are file_conversion.yaml and fileconv/ for conversion. The other syntax definition-helping stuff is in syntax_dev.py and Support and Syntax Definitions subdirs. Only downside is that it’s currently only for ST2 (because I have problems). But well, that’s not in the scope of this thread.

0 Likes

#19

(Off-topic vote for YAML support)

I have been following along to this thread, chiefly because I am working on an improved Markdown syntax. I have gotten used to writing in JSON, chiefly because of the economy of the syntax. But the lack of multiline strings is an issue, as I’d like to annotate my regular expressions before I forget what it is they do :smile:

I have failed (angrily) to use AAAPackageDev in the past and am a happy devotee of PlistJsonConverter – especially the on-save conversion. I would like to see YAML supported in the hope that it might be a more efficient format to work in.

0 Likes

#20

@quodlibet I am considering ripping the yaml portion out and creating a plugin. I mainly use JSON because the plugin was easy to throw together, and it was “good enough”. But if getting yaml module up and running from AAAPackageDev isn’t too bad, this might be a very nice alternative. If I did this, I would construct it similar to the PlistJsonConverter. I just haven’t had the time to look into it yet.

0 Likes

#21

@facelessuser If you were to do this, why not add it to PlistJsonConverter? I mean apart from the name absurdity?

None of my business, how you run your plugin empire, of course :wink:

0 Likes

#22

[quote=“quodlibet”]@facelessuser If you were to do this, why not add it to PlistJsonConverter? I mean apart from the name absurdity?

None of my business, how you run your plugin empire, of course :wink:[/quote]

:smile: The thought occurred to me, and it might actually be done that way in the end.

I don’t think it will be too tough to do. The whole thing is built around PyYaml. It seems pretty easy to get running, and they provide a python 3 and 2 version. There are still some issues I see. Apparently when converting from plist to a python object that can be fed to the yaml converter, plistlib doesn’t use straight up dictionaries, but it uses some class derived from dictionary. I will probably have to iterate through the python object and convert them all over to dictionary classes, but this isn’t a big deal. I am confident I can get a plugin going, you can see it is on its way (at least in a one way conversion form plist to yaml). I literally threw this together by dropping the pyyaml lib into PlistJsonConverter, and deriving a couple new classes from the the language convert class, but you can see it is starting to take shape:

[pre=#282828] begin: ^(("’]?)(repository)(\2))(:slight_smile:
beginCaptures: !!python/object/new:plistlib._InternalDict
dictitems:
‘1’: !!python/object/new:plistlib._InternalDict
dictitems: {name: string.other.quoted-or-unquoted.yaml-tmlanguage}
‘2’: !!python/object/new:plistlib._InternalDict
dictitems: {name: punctuation.definition.string.yaml-tmlanguage}
‘3’: !!python/object/new:plistlib._InternalDict
dictitems: {name: entity.other.repository.yaml-tmlanguage}
‘4’: !!python/object/new:plistlib._InternalDict
dictitems: {name: punctuation.definition.string.yaml-tmlanguage}
‘5’: !!python/object/new:plistlib._InternalDict
dictitems: {name: keyword.control.definition.yaml-tmlanguage}
end: ^(?!\s*($|#))(?=\S)
name: meta.repository-block.yaml-tmlanguage
patterns:
- !!python/object/new:plistlib._InternalDict
dictitems: {include: ‘#comment’}
- !!python/object/new:plistlib._InternalDict
dictitems: {include: ‘#repository-key’}
- !!python/object/new:plistlib._InternalDict
dictitems: {include: ‘#patterns’}[/pre]

I am pretty busy, so don’t expect it in the next couple of days…but keep your eyes out.

0 Likes

#23

Well, you can guess I stumbled with the same problem. I fixed it by mapping the internal Plist dict (no idea why it exists in the first place, it does nothing) to the “yaml.mapping” to the default dict-dumping function (github.com/SublimeText/AAAPacka … #L385-L388).

I also have quite some patching going on and did some work on better dumping of syntax definitions the way I prefer them (multiline dicts only when necessary, use single quotes when useful, specific order of keys …). I even added auto-completion to it. It feels weird to me to read people working on things I already did, and which are open source.

PackageDev doesn’t really have much overhead imo, it defines some useful language definitions for sublime-specific files (settings, keymaps …) and some commands and snippets for these. I’d really like to hear your reasons for avoiding it. If I understand you, I can maybe chat up with guillermoo so that we split the repo into the general utility package it already is and the language def only package.

0 Likes

#24

I have it all working, but I didn’t really look dig that deep or look at what AAAPackageDev did. I mainly looked to see what library was used and ran from there. But I will look into adding your patch which is a bit cleaner…mine was just a quick a dirty fix to keep moving.

The PackageDev yaml syntax is very nice for sublime stuff, but still has some things to be hammered out. None of these things are complaints really, just noted stuff and opinions. First off, you have done amazing considering how much of a pain it is to parse YAML…parsing YAML with regex sucks. I don’t think there can ever be a perfect YAML highlighter in sublime just because regex alone just doesn’t cut it for YAML. Anyways, PackageDev is way too strict on keys (but I get what you are trying to do); repository keys all get invalid highlighting. Oniguruma regex syntax seems to cause highlight failures on occasion; falling back to a simpler regex highlighter gave overall better results. Obviously there are more issues than that, but like I said, YAML sucks to highlight. Sublime Text’s default YAML is horrible.

I got the plugin all working (needs cleanup), and I threw together a more simple stripped down YAML language file which is better than sublimes, but not as sublime specific or as exhaustive as yours. But as a compromise for YAML in general, I saw the ajax.org ace project had some decent rules, so I grabbed those and have been building on that…oh and it has its own issues as well though :smile:. I think if I can just get the comments to be little better, I will be happy enough for now.

I will be opening a beta thread soon so I can leave this thread alone and stop getting it off topic :smile:.

YAML generated from PLIST:
[pre=#1E1E1E]fileTypes:

  • yaml
    name: YAML (simple)
    patterns:
  • match: \t+
    name: invalid.illegal.tab.yaml
  • match: ‘#.*$’
    name: comment.yaml
  • captures:
    ‘1’:
    name: markup.list.yaml
    match: ^(-{3}|.{3})\s*(?=#|$)
  • captures:
    ‘1’:
    name: markup.list.yaml
    match: ^\s*(-?])(?:$|\s)
  • match: ‘!!\w//]+’
    name: constant.yaml
  • match: ‘&*][a-zA-Z0-9-_]+’
    name: constant.language.yaml
  • captures:
    ‘1’:
    name: meta.tag.yaml
    match: ^\s*(\w.*?)(:(?:\s+|$))
  • captures:
    ‘1’:
    name: meta.tag.yaml
    match: (\w+?)(\s*:(?:\s+|$))
  • captures:
    ‘1’:
    name: meta.tag.yaml
    ‘2’:
    name: string.yaml
    match: ‘\s*("]((?:(?:\.)|(?:^"\]|\"))*?)"])(:(?:\s+|$))’
  • captures:
    ‘1’:
    name: meta.tag.yaml
    ‘2’:
    name: string.yaml
    match: ‘\s*(’’]((?:(?:\.)|(?:^’’\]|’’{2}))*?)’’])(:(?:\s+|$))’
  • match: <<\w*:\w*
    name: keyword.operator.yaml
  • match: -\s*(?={])
    name: keyword.operator.yaml
  • match: ‘"](?:(?:\.)|(?:^"\]|\"))*?"]$’[/pre]
0 Likes

#25

I beg you, please report any issues you have with the PackageDev YAML language (preferrably on the repo’s issues page) with some sample code. I built the syntax definition completely on my own, from my own examples. Since I whitelisted the keys (in order to highlight rules in a certain way) I just used what I got (I also forced an indent of at least 2 spaces, which is recommended anyway). Whenever a definition broke highlighting, e.g. when I converted a different language into YAML, I adjusted it, but I can’t fix something I don’t know about or that never happened to me. Especially the oniruguma regex thing is interesting because this should work rather well.

If github supported some kind of messaging system, that would be cool. We could also use PMs on this forum, but I fear that I might not notice an unread PM. I think continuing this on issue(s) is an overall better idea than to keep polluting this thread with OT. If everything fails, you are free to send me an email (to my github’s address).

0 Likes

#26

Yeah, I will post some issues sometimes soon.

0 Likes