Home Download Buy Blog Forum Support

New SFTP plugin needs testers: Update: Linux and OSX support

New SFTP plugin needs testers: Update: Linux and OSX support

Postby CodeEffect on Tue May 28, 2013 10:02 am

EDIT: UPDATE, Linux and OSX support added as of the 16th June :)

I'm afraid it's windows only and SFTP only for now (it's unlikely to ever support FTP). It uses psftp and plink to connect to servers and because it uses plink there are always likely to be issues with different app versions (ls, grep, etc) supporting (or not) different command line switches. I'm throwing this version out there to get a bit of exposure to these issues and hopefully make it as resillient and robust as possible with as little configuration from the end user as it can get away with.

It's only just over a week old so should be considered VERY alpha software. Please make sure you have good backups of anything that you let it near. Please don't use it on production *anything*. I am not responsible if it eats your dog and slaps your mother with a fish etc etc etc, you know the score :)

To make file browsing and the equivalent of CTRL+P as fast as possible RemoteEdit can index your server. It does this in quite a simple way by piping the output from a recursive ls to a file then pulling it down and parsing it. This obviously has serious security implications and should only be considered for development servers where it doesn't matter if the contents were to be accidentaly shared or lost. This functionality is enabled by setting enable_cat and cat_path in the server setting file.

At the moment all functionality is available via the F5 keyboard shortcut.

Known bugs:
- All the todo's in the RemoteEdit.py ;P
- The order in the server list isn't set correctly so clicking on server 1 may get you server 3, server 2 may get you server 1, etc etc. Only just noticed this this morning so I'll get it fixed tonight.
- Another one found this morning: Grep "--exclude-dir" isn't universally supported. It is on Debian 6, it isn't on CentOS release 5.9 (Final).

If you do wish to have a play with it and help out with bug repots please could you give as much information as you have available. Ideally I would like to know the relevant lines of debug output from the console, the flavour and version of the remote OS you're running (e.g. Debian Squeeze, Gentoo latest etc). Please also keep it to this thread rather than raising issues on github, at least for now.

If you have feature requests feel free to note these here too. They may not get a timely individual response for the moment but I will have noted them all down and will consider them when I get time.

I'm now at work so won't be able to really respond to a lot until this evening or even possibly later in the week. If you get stuck the console logs quite a bit of useful info. I'll try to help out where I can.

EDIT: Would help if I posted the URL LOL: https://github.com/CodeEffect/RemoteEdit
Last edited by CodeEffect on Sun Jun 16, 2013 12:43 am, edited 2 times in total.
CodeEffect
 
Posts: 20
Joined: Mon May 06, 2013 6:18 am

Re: New SFTP plugin needs testers

Postby robertcollier4 on Tue May 28, 2013 1:38 pm

Running Windows XP SP3 32-bit ST3 Build 3033. Console output:

Code: Select all
Process dead or never started
Waiting for output...
---------- OUT ----------
Linux servername 2.6.32-5-amd64 #1 SMP Mon Feb 24 12:26:11 UTC 2013 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

]0;user@servername: ~user@servername:~$

Sending command: ls -lap --time-style=long-iso --color=never "/home/username/"
Waiting for output...
---------- OUT ----------
ls -lap --time-style=long-iso --color=never "/home/username/"

total 60K
drwxr-xr-x 6 username username 4.0K 2013-04-19 22:38 ./
drwxr-xr-x 4 root root 4.0K 2013-04-02 23:10 ../
drwx------ 2 username username 4.0K 2013-04-03 13:22 .aptitude/
-rw------- 1 username username 8.0K 2013-05-28 09:03 .bash_history
-rw-r--r-- 1 username username  220 2013-04-02 23:10 .bash_logout
-rw-r--r-- 1 username username 3.5K 2013-04-03 13:44 .bashrc
drwxr-xr-x 2 username username 4.0K 2013-04-03 13:59 bin/
-rw------- 1 username username  827 2013-04-04 05:15 .mysql_history

]0;username@servername: ~username@servername:~$

Traceback (most recent call last):
  File "D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py", line 220, in handle_server_select
    self.start_server(self.items[selection - 2])
  File "D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py", line 265, in start_server
    self.open_server()
  File "D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py", line 279, in open_server
    s = self.list_directory(self.lastDir)
  File "D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py", line 1542, in list_directory
    self.lastDir
  File "D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py", line 1925, in parse_ls
    s = int(sl[4])
ValueError: invalid literal for int() with base 10: '4.0K'


I think the above is caused because it is defined in my user's bash.rc (this is commonly defined as default in many ubuntu installs):
Code: Select all
alias ls='ls --color=auto -alh'


from man ls
Code: Select all
-h, --human-readable
              with -l, print sizes in human readable format (e.g., 1K 234M 2G)
robertcollier4
 
Posts: 153
Joined: Sun Feb 24, 2013 5:37 pm

Re: New SFTP plugin needs testers

Postby CodeEffect on Tue May 28, 2013 7:31 pm

Cheers for that, I think you've got it. Shouldn't be too tricky to parse that back into bytes (heheh and then back into KB, MB, GB etc for display later :) ).

I'll have a look a bit later and update here when it's done.

EDIT: Fix to that issue (and the server order one) should be in commit: https://github.com/CodeEffect/RemoteEdi ... bf5fd4a9cc
CodeEffect
 
Posts: 20
Joined: Mon May 06, 2013 6:18 am

Re: New SFTP plugin needs testers

Postby CodeEffect on Sat Jun 01, 2013 8:23 pm

All ssh / sftp communication code has now been moved into a separate module and we communicate with it via a queue.

A ton of other small updates have been done as well. It's really beginning to come together now and is robust and reliable enough for me to use it at work.

If you've got ST3, a need for editing files over SFTP and you're running windows I could really use your feedback :)
CodeEffect
 
Posts: 20
Joined: Mon May 06, 2013 6:18 am

Re: New SFTP plugin needs testers

Postby robertcollier4 on Sun Jun 02, 2013 5:17 am

Even with the new version, while it is connecting or getting files or doing anything over the network, my SublimeText often freezes and becomes unresponsive for 5-10 seconds. The network communication needs to be "non-blocking" to SublimeText.
robertcollier4
 
Posts: 153
Joined: Sun Feb 24, 2013 5:37 pm

Re: New SFTP plugin needs testers

Postby robertcollier4 on Sun Jun 02, 2013 4:42 pm

This might be helpful:
https://code.google.com/p/pysftp/
robertcollier4
 
Posts: 153
Joined: Sun Feb 24, 2013 5:37 pm

Re: New SFTP plugin needs testers

Postby CodeEffect on Sun Jun 02, 2013 5:27 pm

Yep, am moving towards that. At first it was all inline (while I was trying to get things working), now parts of it are sort of event driven (the cataloging happens in the background now), once I've tidied things up the plan is for it to be all completely events driven. Most of the groundwork has now been done, it's just a case of plumbing it all in. Should be fixed in the next few days.

The sftp functionality now works by putting your commands onto a queue and waiting for the results to come back. You can pass a callback with your command and effectively forget about things until the results come back. At the moment the results checking works by setting a timeout and checking for the results, if not there then set a timeout, check for results etc etc. My next task is to remove the timeout bits and have it be completely event driven.

Cheers for the link but I don't think I can use pysftp as paramiko doesn't support python 3 (as best I can tell) :( Would love to have it all python without needing to use psftp.exe but I don't think it's possible without writing the sftp code at the moment. The sftp part is now its own module so it should be relatively trivial to swap out when something is available. Same applies to getting it working on different platforms, should just be a case of writing the sftp module for that platform.
CodeEffect
 
Posts: 20
Joined: Mon May 06, 2013 6:18 am

Re: New SFTP plugin needs testers

Postby CodeEffect on Wed Jun 05, 2013 3:52 pm

Good to go I think :D

Most of the network calls have now been made non-blocking. My next task will be to re-factor things a bit and tidy up the code. Would be good to get some bug reports in for platforms where it's not working as it should.
CodeEffect
 
Posts: 20
Joined: Mon May 06, 2013 6:18 am

Re: New SFTP plugin needs testers

Postby neema on Thu Jun 06, 2013 3:18 pm

I am running into some trouble when selecting a remote file. Console output:
Code: Select all
MAIN 10:30:59: Results found in callback handler, firing the callback
MAIN 10:30:59: Trying cat for folder "/work/me/src/tools/ax_lua/"
MAIN 10:31:02: root: /work/me/src/tools/ax_lua, selected: ax_lua.c
MAIN 10:31:02: run_remote_command called for sftp and cmd: "get /work/me/src/tools/ax_lua/ax_lua.c C:\Users\me\AppData\Local\Temp\RemoteEdit\Buzz\work\me\src\tools\ax_lua\ax_lua.c"
MAIN 10:31:02: ....now on the queue.....
MAIN 10:31:02: Using set_timeout to call the callback handler to check for results
SFTP[0] 10:31:02: Process not running: 'NoneType' object has no attribute 'poll'
SFTP[0] 10:31:02: Waiting for output...
SFTP[0] 10:31:04: --------- stdout ---------
Remote working directory is /home/me
psftp>
--------------------------------------------
SFTP[0] 10:31:04: Waiting for output...
error: Error downloading ax_lua.c


if I run the get command manually through the included psftp.exe exactly as above the file downloads immediately. Any thoughts?


*Edit: I am connecting to an AIX unix box, as I said the psftp.exe works fine and I use winscp regularly. One note about platform support, your ls command assumes gnu ls so I had to manually change the lsParams.
neema
 
Posts: 1
Joined: Thu Jun 06, 2013 3:14 pm

Re: New SFTP plugin needs testers

Postby CodeEffect on Thu Jun 06, 2013 5:43 pm

Hi, thanks for the feedback.

Firstly, with regard to the bug you're seeing I've a feeling it could be fixed by the updates I have waiting to commit. I've just got to tidy up 1 more method and it's ready to go in. I'll get this in as soon as I finish work (next few hours) and ping you a PM. If the updates don't fix it though, could I ask fo a bit more console output please? It's very, very verbose at the moment.

WRT BSD, good spot. Am pulling down an ISO for freebsd at the mo to do some testing against, it's been too long!

I've been pondering over the best solution for handling different env's and I now have a plan:
On first connect stay in SFTP mode until we can do some feature detection. Once the user has finished what they are doing then fire off some tests to see what functionality is supported by ls and grep. As soon as these are done we'll save results back to the server config file so it doesn't have to run them again. Perhaps these could be manual settings in the server config file but if you weren't sure then RE will automatically fill in the blanks. As soon as blanks have been filled it can go into full SSH mode and fire off the catalogue too. This way it should *just work* for everyone from the start although for the first minute or two some features may not all be available.

EDIT: Found the bug you were experiencing and fixed it now :)
CodeEffect
 
Posts: 20
Joined: Mon May 06, 2013 6:18 am

Next

Return to Plugin Announcements

Who is online

Users browsing this forum: No registered users and 8 guests

cron