Syq - Fast programmable file operations across machines
Show HN: Syq – copy files between machines fast (better than rsync)
Syq is a fast, programmable file transfer tool that outpaces rsync for copying, reorganizing, and removing files locally or across machines. It lets you resume interrupted transfers, send files to your laptop or run commands there from a server's remote shell without needing an SSH server or open port on your laptop. You can script syq using its JSON API or Python SDK, copy directly between servers, and rename or reorganize files during a copy. If you already use rsync, just prefix your command with syq to get started.
Your laptop needs no SSH server or incoming network port.
- hxseven
I thought I'd share a more positive comment. I work on a tool in a similar area (moving stuff around) but on a higher level. I currently use rsync or rclone for the file moving parts. So to me your tool looks interesting and promising, thanks for sharing it :)
Some suggestions for improvement for your website/Github:
- I wanted to understand how it differs from rsync. A comparison page would be helpful.
- A comparison to rclone would also be interesting, as it also uses multiple parallel connections.
- As it target more advanced users I think it would be helpful to provide more advanced technical insights on how your transfers exactly work. Also how are edge cases handled, what tests are done? Moving and copying files is critical, and you need to inspire trust in your project among users.
- unsnap_biceps
> I also added cool features like the ability to maintain a persistent ssh connection to the server for fast one-offs
rsync already supports this via OpenSSH's ControlMaster directive. Bonus, it speeds up every connection to that server rather then a single tool's.
> the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop).
You can control your agent forwarding in your local ssh client configuration. And with modern ProxyJump, you don't need to forward your agent at all regardless of how many bastion hops are between you and your final target.
- jttnr
Open page, read headline, read first sentence, emdash, close page.
- AceJohnny2
Thank you for showing this. I'll echo what some others say: if you're claiming to be faster than rsync, you should demonstrate why front-and-center. There have been decades of documentation on how rsync optimizes transfers, so it's a lot to live up to.
As an aside, It's perhaps an indictment of our networking landscape that multiple parallel connections between 2 specific machines would accelerate a transfer. I would've expected a single TCP connection to be able to saturate a line. Or perhaps the parallel connections seeks to amortize the per-file setup overhead?
- hikarudo
Very cool!
How does the direct TCP mode work?