Home Download Buy Blog Forum Support

how to join lines in huge file very fast?

how to join lines in huge file very fast?

Postby nardev on Sun Dec 08, 2013 11:37 pm

I have to join lines in very huge files very often, Ctrl + J works but it takes about 10 or more seconds in my case. Because files are huge. Even for smaller files with 500 lines.

Is there any faster way to join all lines?

p.s.
Structure of files is usually HTML/JS/CSS
nardev
 
Posts: 7
Joined: Thu Mar 15, 2012 5:34 pm

Re: how to join lines in huge file very fast?

Postby iamntz on Mon Dec 09, 2013 5:45 am

Regex search for \n and hit replace with the replace box empty.
iamntz
 
Posts: 910
Joined: Fri Apr 29, 2011 8:52 am
Location: Romania

Re: how to join lines in huge file very fast?

Postby oldwarhorse on Mon Dec 09, 2013 7:52 am

nardev wrote:Is there any faster way to join all lines?

Yes.
iamntz wrote:Regex search for \n and hit replace with the replace box empty.

Nope. That's not it.
I think that's what Ctrl+J is doing anyway, b/c on a 100k js file, ~3500 lines, both take the same amount of time, over 10 seconds, and peg ST3 CPU at 25% for me. (Win7, 3ghz, 6gb). It must be re-writing the whole string every time, after every line.

Nope, pretty sure you're gonna have to make yourself a "StringBuilder". That's the fastest way I know of to do it.
For instance, the following simple demo code reads in the 100k file, splits it on linesep, and re-joins it with empty strings, and writes it out.
It runs in about .5 seconds.
Running as a plugin would be even faster, b/c you'd have no device I/O time. You'd just be using the buffer contents.
Here's the code (Python 3.3.3):
Code: Select all
import os
os.chdir('C:/users/dave')
f = open('C:/NodeJS/Apps/xxx_ServerX/build/xxServer/xxServer.js', 'r')
fil = f.read()
f.close()

#As a plugin, just get the active window buffer contents into a string variable...
filSplit = fil.splitlines()                     #<---    These 2 lines are what you need.
filJoined = ''.join(filSplit)                  #<---   ( '' ) is 2 single-quotes, not a double-quote
#then write it back to the buffer

f = open('jointest.txt', 'w')
f.write(filJoined)
f.close()
print("Done.")

I did a package search for something that would do this and did not find anything, but I did find this:
https://sublime.wbond.net/packages/StringUtilities
If you're not up on writing plugins, (and I'm not, at the moment - been too long), then you might be able to just install that utility and then kludge the 2-line "StringBuilder" code into it, (by copying one of the other functions in the utility), and have yourself a quick out.
(Unless anyone else knows of one.)

All that being said, you do know there are components/libraries like combo-izers and minifiers that already do this kind of thing for you, right?
For instance, I use "shifter" to minify JS files... https://github.com/yui/shifter
That sort of thing.

Hope this helps.
Dave
oldwarhorse
 
Posts: 40
Joined: Wed Oct 03, 2012 12:34 am


Return to Technical Support

Who is online

Users browsing this forum: No registered users and 15 guests