Sublime Forum

SLT Freezes at 100% CPU with JSON syntax

#1

Hi there,

The file in question is a copy&paste saved from the JIRA REST API call “rest/api/2/issue/createmeta”, it’s a single line of JSON, about 500kB in length. As far as I can tell, the file is valid.
When I switch the syntax to JSON or load the file as JSON, SLT2 will freeze and utilise 100% (or as much as it can get) from one CPU.

Anyone got any ideas or fixes?

System is:
OSX 10.9 (dev preview); most recent stable SLT 2

0 Likes

#2

I solved this by adding a newline before every “{”.

However, this action was not possible in Sublime Text either, even with “Text” syntax scheme – SLT would just freeze up.

0 Likes

#3

I’m not sure if Sublime has any particular line length limit, but thought I would do some tests here:

  1. Took my user prefs JSON file and concatenated it onto one line. Then grew it up to over 560k by repeating its content, ensuring it’s syntactically correct.
  • ST2: Remains responsive, although minor slowdown in the UI. Syntax colouring remains active
  • ST3: Remains responsive, although minor slowdown in the UI. Syntax colouring deactivates (somewhere around 16k length)
  1. Doubled the JSON line length to over 1.1M
  • ST2: Remains usable though less responsive. Stalls when holding down Up/Down or PgUp/PgDown. Minimap and scrollbars usable. Text modification is laggy but usable. Syntax colouring remains active. Switching syntax to “Plain Text” makes no difference to performance. No crash or permanent freeze.
  • ST3: Remains usable though less responsive. Stalls when holding down Up/Down or PgUp/PgDown. Minimap and scrollbars usable. Text modification is laggy but usable. Syntax colouring not active. Switching syntax to “Plain Text” makes no difference to performance. No crash or permanent freeze.
  1. For shits and giggles, I tried cranking the line length to 18M
  • ST2: Very unresponsive UI, takes close to 2 seconds to respond to cursor movements. Same with PgUp/Down etc. Edits take 4-5 seconds. Colouring remains active.
  • ST3: Very unresponsive UI, takes close to 2 seconds to respond to cursor movements. Same with PgUp/Down etc. Edits take 4-5 seconds. No syntax colouring.

So, in essence, I’m seeing the same behaviour for long lines in JSON in both ST2 and ST3 except that ST3’s colouring switches off at some length, around 16K I think. Interestingly, colouring being on in ST2 for huge lines doesn’t seem to degrade performance at all, at least for basic navigation and editing.

When line length exceeds 500K, Sublime’s UI starts to struggle increasingly. It remains comfortably usable at under 1M, and worsens exponentially after that. View…word wrap being on or off makes no difference to the above, nor does switching syntax to “plain text”.

Summary:

Though Sublime remained stable throughout these simple tests, it felt increasingly flaky with longer lines. It would be nice to see such performance issues addressed. Testing took place on a pretty powerful (3Ghz quadcore) machine (using x64 sublime); I’d expect things to be worse on a typical Core2/i5 type laptop.

Quite often as programmers we need to open big text files and perform manipulations upon them. They won’t always start out in a “nice” (linebroken) format to begin with; ironically we may want to load them up in Sublime to make them so. Such file and line sizes (up to a few Meg) are small and not uncommon these days.

I also noticed a lack of multiprocessing going on in these tests. All the CPU load was maxing out a single core. However Jon has engineered the editing core of Sublime, it could use some improvements.

0 Likes

#4

Good job,I appreciate your work.Thanks for sharing own good work.

0 Likes

#5

I believe part of the issue is that Sublime Text uses language syntax definitions that are based on regular expressions, so it depends on how efficient the regular expressions for a given language are. I’ve definitely noticed slowdowns opening files with 50K lines and changing the syntax to SQL and using 50K cursors to do batch edits. Then again, I don’t expect good performance since that is kind of an insane number of operations being done with each keypress.

I would imagine with longer files it may be possible to highlight the file more incrementally, but it may not be optimized for longer lines. Additionally, I imagine the regular expression engine used would need to support multiple cores in order to see such usage. I doubt Jon wrote his own regex engine, although it isn’t out of the realm of possibility.

0 Likes

#6

[quote=“wbond”]I believe part of the issue is that Sublime Text uses language syntax definitions that are based on regular expressions, so it depends on how efficient the regular expressions for a given language are. I’ve definitely noticed slowdowns opening files with 50K lines and changing the syntax to SQL and using 50K cursors to do batch edits. Then again, I don’t expect good performance since that is kind of an insane number of operations being done with each keypress.

I would imagine with longer files it may be possible to highlight the file more incrementally, but it may not be optimized for longer lines. Additionally, I imagine the regular expression engine used would need to support multiple cores in order to see such usage. I doubt Jon wrote his own regex engine, although it isn’t out of the realm of possibility.[/quote]

There is certainly a performance penalty in using complex regex’s in real time to perform syntax colouring; my guess is that Sublime uses python’s native regex facilities. However I don’t think that’s the issue here. The problems above:

  • Concern long lines, not big files per se
  • Happen irrespectively of whether we’re in “plain text” syntax. Performance issues relating to syntax colouring are typically addressed by switching to plain text (no colouring)
  • Are irrespective of whether any kind of wrapping is happening
  • Happens in ST2 and ST3, with performance degradation similar in both; ST3 imposes a line length threshold for disabling colouring, whereas ST2 doesn’t. Yet ST3 sees no performance improvement over ST2 as line length increases, even beyond the threshold - see my post above.

While there are definitely performance issues connected with colouring algorithms and regex’s, this issue is likely down to the way ST handles text internally. What I’d like to see:

  1. Some improvements in ST’s core for handling large files and long lines. Most editors handle this fine and Sublime shouldn’t be the exception. At the moment a lack of threading combined with what feels like clunky long line behaviour is a PITA and requires me to fire up other editors for handling big mysql dumps etc…
  2. The ability to fine tune or disable the line-length threshold at which ST3 turns off colouring. ST2 doesn’t have a limit that I’ve seen, and there’s no performance drop with colouring enabled on really long lines in ST2.
  3. (related, nice to have) A high-priority UI, perhaps on the status bar, that provides a progressbar and “cancel” button for any time consuming operations. Sublime can freak me out when it becomes totally unresponsive working on a larger file and I do a particular SnR or some other action. I’ve no way to know beforehand the actions that will lock up the UI and those that’ll be quick. Ideally the UI should always be responsive, even if the core is tied up doing something, at least for progress display and cancel option.
0 Likes