Sublime Forum

Dev Build 3085

#1

Dev Build 3085 is out now. The majority of the work in this release has gone into writing a new regex engine, to partially replace Oniguruma. While it is substantially faster, it doesn’t support all of the Ongiuruma functionality, and falls back to Onig where necessary. Depending on file type, the speedup can be significant or minor, depending on what sort of regexes the syntax definition is using. I’m planning on getting more regexes on the fast path in the future.

1 Like

#2

I tried on a large file (3Mo) : went from from ~7sec to ~3sec. Pretty good improvement :smile:

0 Likes

#3

Jon,

Thanks for the recent updates! One recent issue, though. Did folding break in HTML files in build 3085, or is it just me?

I’m running Mac Yosemite.

Thanks,
Paul

0 Likes

#4

Any plans to add API support for the new Changelog window? That would be a great addition!

0 Likes

#5

I see a bunch of logs in the console with no reference to a file, how can I fix them if I don’t know which syntax is causing them:

unexpected rule type, was expecting one of match, include, begin or patterns, got: {comment: Note how all @rules are prefixed.}
unknown include #string-double
unknown include #string-single
unknown include #string-single
unknown include #string-double
unknown include #media-query-list
unknown include #media-query-list
0 Likes

#6

Since the update to Sublime Text build 3085, the folding doesn’t work as well the code indenting doesn’t work anymore on my machine.
I am running the latest version of Mac OS X Yosemite.

0 Likes

#7

Thanks for the new update. But seems the previous builds’ bug/regression not fixed yet.

They are:

[quote=“singwong”]@jps, there’s another reloading Network File (NFS) bug.
I am using build 3082 (also 3084). When I opened a network file , I do not edit it, and someone edited it at other computer afterward, it should be silently reloaded at my side (from my experience on Sublime Text 2), but the file do not reload, and when I do some changes and save it on my side, it overwrite the changes saved by others with no prompt.
I already set “always_prompt_for_file_reload”: false (same as Default) in my User Setting file

Till to this bug, I overwrote 3 times of others changes already. :cry:[/quote]

[quote=“singwong”]Seems there a regression in build 3084, I can use “StringEncode: URL Decode” (packagecontrol.io/packages/StringEncode) command in build 3082 to decode a url string, but after upgraded to 3084, it failed and show the following traceback at console:

Traceback (most recent call last): File "/Applications/Sublime Text.app/Contents/MacOS/sublime_plugin.py", line 556, in run_ return self.run(edit) File "string_encode in /Volumes/Data/MacSystem/Users/sing/Library/Application Support/Sublime Text 3/Installed Packages/StringEncode.sublime-package", line 33, in run File "string_encode in /Volumes/Data/MacSystem/Users/sing/Library/Application Support/Sublime Text 3/Installed Packages/StringEncode.sublime-package", line 165, in encode File "./urllib/parse.py", line 615, in unquote_plus TypeError: expected bytes, bytearray or buffer compatible object[/quote]

0 Likes

#8

Thanks for the report - something is off with regards to code folding and tabs, will fix it in the next build

0 Likes

#9

Thanks for the variables addition, can’t wait to test it in my YAML definition.

I propose adding a hyphen to the allowed characters for a name (or actually all characters but {, ``, ] and }).
The only two situations where two literal opening braces may occur are fairly limited. 1) is inside a set, which hardly makes sense so shouldn’t be an issue in terms of accidentally triggering a substitution and 2) immediately atfer an escaped opening brace (to repeat it several times), i.e. \{{4}, in which case you can not append a second closing brace since it’s invalid syntax (has to be escaped).

0 Likes

#10

Hi, I would like to suggest a compatibility option to explicitly chose the old regex engine on the bundle configuration. Please see this issue: github.com/davidrios/jade-tmbundle/issues/47. Thank you.

0 Likes

#11

You are probably going to have to convert your theme to a sublime-syntax version and update accordingly to fix any issues. You can also provide an old tmLangauge file which things like ST2 and old ST3 versions would use as they won’t recognize the new sublime-syntax format. This way you can support both. When the next ST3 releases on the beta channel, you can probably create a separate branch for either ST2 or ST3 and remove the legacy format for ST3. With Package Control you can then release separate releases for ST2 and ST3. As ST3 is the future, you can decide how long you want to support both branches.

Now this may require work on your part, but things are changing. ST3 is a going to be a new major release. I doubt Jon is going to want to leave in the old legacy regex engine to bloat the program, and frankly devs are going to have plenty of time to update packages before the final ST3 release comes. I guess if full support for the old engine is still there, maybe he will expose a way to use it, but I imagine he eventually wants to kill off any fallbacks to the old engine and maybe remove it, but only he could tell you for sure. I think he mentioned allowing users in the future to contribute to the default languages; I imagine they will all eventually be moved over to this new format making the old obsolete.

Anyways, long story short, you can currently support older Sublime Versions with the tmLangauge and support the new engine with sublime-syntax, but may require some extra work on your part.

0 Likes

#12

In sublime-syntax, this works:

(?imx-imx)         option on/off

But this does not work:

(?imx-imx:subexp)  option on/off for subexp

This keeps a good number of syntax from directly converting. I really feel like this should be allowed.

0 Likes

#13

@facelessuser I understand that ST3 is the future of Sublime Text, but it is not the future of everything. There are other things that use the tmLanguage format, notably TextMate where the format comes from, and now GitHub also uses it for its syntax highlighter. Are you saying then that Sublime is explicitly breaking compatibility with TextMate?

0 Likes

#14

I get that. I am not going to pretend to know the mind of Jon and what he will or will not support. If I were just supporting Sublime, I would migrate to the new format. The whole point to writing a new syntax engine was to improve things, I don’t know if he did it because he wants to replace the old system, or support two systems.

If you want a package that supports all editors that mimic the TextMate engine, you may have a unique problem if Jon is moving away from TextMate support. If he means to support both, then I would I would be very specific to what TextMate compatibility features are now broken; right now you have left it kind of vague and opted to just suggest directly using the old support instead of clearly defining what is broken; Jon may not even be aware of what is broken. He may not have even provided a way to specifically utilize just the old method. The sublime-syntax method was meant to be a separate language syntax from the tmLangauge support. And the rewriting of the parts of the old engine were meant to boost speed as well, but not necessarily break tmLanguage support (from what I understand).

So if you can be specific to what is broken, you may get it fixed in one of the next releases. I am not sure what this is referring to and I am not sure Jon knows either:

and since this highlighter rely on some obscure features of the old engine

What obscure features are not working? Sublime TextMate support has never been 100%, but close enough,so I don’t know if Jon ever used these obscure features.

0 Likes

#15

I’m just going to quote this:

Basically, functionality should remain the same and only be faster in most occasions while falling back to the old legacy engine that is Oniguruma in this case when the new one does not support a feature.

Meaning: Every syntax definition that is not working anymore in 3085 is a result of a regression and a bug that should be fixed since the feature set is supposedly equal, as far as I’m concerned.

@facelessuser: Instead of (?imx-imx:subexp) you can try (?:(?imx-imx)subexp). It should be the same functionality-wise but I don’t know how it’s implemented. Still a bug.

0 Likes

#16

[quote=“FichteFoll”]
@facelessuser: Instead of (?imx-imx:subexp) you can try (?:(?imx-imx)subexp). It should be the same functionality-wise but I don’t know how it’s implemented. Still a bug.[/quote]

Thanks. I don’t have a problem working around it, but I have to manually do it when I run the convert tool which kind of sucks. And if you don’t know what the issue is, it isn’t obvious at first. I don’t particularly have any use cases where I am applying one of these to just a group, but in general, many tmLanguages do this even when they don’t have to causing convert issues. It would be nice if the root issue just gets fixed, but yeah, it is also good to post the workaround for others as well.

0 Likes

#17

Have any of you experienced any performance differences between using a regex with a bunch of keywords vs doing each one written out?

\b(?i:AND|OR|NOT|TRUE|FALSE|SELECT|INPUT)\b

vs.

\b(?i:AND)\b
\b(?i:OR)\b

I have about 400 keywords I need to include in a syntax def I am creating the load time is brutal. It’s still better than the tmLanguage format but still not as fast as I would like.

Here is the main context where each include is a list of keywords that needs to be checked. Any suggestions to make this faster would be helpful. I’ll be doing my own performance comparisons this weekend, but wanted to post here to see if anyone has any better ideas.

contexts:
  main:
    - include: comments
    - include: compOperators
    - include: mathOperators
    - include: types
    - include: functionDefs

    - include: keywordsControl
    - include: keywordsSql
    - include: keywordsOther
    - include: keywordsOtherUnsorted
    - include: keywordsString
    - include: keywordsBool
    - include: keywordSupport

    - match: \b(?i:TRUE|FALSE|NULL|NOTFOUND)\b
      scope: support.constant.4gl

    - match: \b\d+\b
      scope: constant.numeric.4gl

    - match: '"'
      push: string_double

    - match: \'
      push: string_single

    - match: \(
      push: parens
    - match: \)
      scope: invalid.illegal.stray-bracket-end

    - match: \
      push: brackets
    - match: \]
      scope: invalid:illegal.stray-bracket-end

Thanks!

0 Likes

#18

Try atomic groups, they are better in performance since there will be no useless backtracking. Should be fine for plain keyword matching.

0 Likes

#19

Any incompatibilities with Oniguruma are bugs, and will be fixed. The new regex engine is intended to be entirely invisible, aside from any efficiency gains.

I’ll fix the issues mentioned in the above ticket.

0 Likes

#20

[quote=“facelessuser”]In sublime-syntax, this works:

(?imx-imx)         option on/off

But this does not work:

(?imx-imx:subexp)  option on/off for subexp

This keeps a good number of syntax from directly converting. I really feel like this should be allowed.[/quote]

It should work, and does in the limited set of tests that I’ve got. Do you have an example of where it’s not working?

0 Likes