Sublime Forum

How to join lines in huge file very fast?

#1

I have to join lines in very huge files very often, Ctrl + J works but it takes about 10 or more seconds in my case. Because files are huge. Even for smaller files with 500 lines.

Is there any faster way to join all lines?

p.s.
Structure of files is usually HTML/JS/CSS

0 Likes
#2

Regex search for \n and hit replace with the replace box empty.

0 Likes
#3

Yes.

Nope. That’s not it.
I think that’s what Ctrl+J is doing anyway, b/c on a 100k js file, ~3500 lines, both take the same amount of time, over 10 seconds, and peg ST3 CPU at 25% for me. (Win7, 3ghz, 6gb). It must be re-writing the whole string every time, after every line.

Nope, pretty sure you’re gonna have to make yourself a “StringBuilder”. That’s the fastest way I know of to do it.
For instance, the following simple demo code reads in the 100k file, splits it on linesep, and re-joins it with empty strings, and writes it out.
It runs in about .5 seconds.
Running as a plugin would be even faster, b/c you’d have no device I/O time. You’d just be using the buffer contents.
Here’s the code (Python 3.3.3):

import os
os.chdir('C:/users/dave')
f = open('C:/NodeJS/Apps/xxx_ServerX/build/xxServer/xxServer.js', 'r')
fil = f.read()
f.close()

#As a plugin, just get the active window buffer contents into a string variable... 
filSplit = fil.splitlines()                     #<---    These 2 lines are what you need.
filJoined = ''.join(filSplit)                  #<---   ( '' ) is 2 single-quotes, not a double-quote
#then write it back to the buffer

f = open('jointest.txt', 'w')
f.write(filJoined)
f.close()
print("Done.")

I did a package search for something that would do this and did not find anything, but I did find this:
sublime.wbond.net/packages/StringUtilities
If you’re not up on writing plugins, (and I’m not, at the moment - been too long), then you might be able to just install that utility and then kludge the 2-line “StringBuilder” code into it, (by copying one of the other functions in the utility), and have yourself a quick out.
(Unless anyone else knows of one.)

All that being said, you do know there are components/libraries like combo-izers and minifiers that already do this kind of thing for you, right?
For instance, I use “shifter” to minify JS files… https://github.com/yui/shifter
That sort of thing.

Hope this helps.
Dave

0 Likes