Production Use and Client Source

I love this product, there has long been a need for such a simple way to backup windows machines (backuppc has tried but too much PT in getting the windows side to work correctly [open files et al]). So great job. I want to roll this out as my standard backup system for my clients but its missing one or 2 features. I am quite happy to add these features myself, but I can’t seem to find the source to the client, can I get my hands on that please.

What I would like to add are

  1. Change the PC backup label (Allow me to override the use of the PC’s windows machine name as the label for backups of this machine)
  2. Ability to set the label of the backup path. I see that you can do this by editing the .db file using an SQL lite editor and while thats all well and good, I would like to make it simpler.
  3. Ability to filter out file types per backup path, so mydocuments, I don’t want to backup .mp3 files or .avi files for when users download all their movies and personal songes (>lots of GBs) and stick them in their mydocs, we can avoid backing them up.

That would really sort it for me, is the client app open source as well?

The client code is interleaved with the server code in the “backend” git repository. The Tray icon code is in “frontend_wx”.
Changing the pc label is actually quite hard if you want to do it on the server side. On the client side it’s easy: urbackup/client.cpp:161
I didn’t get what you mean by label of a backup path? Do you mean the general backup directory where backups are saved in or specific client directories?
The third thing could be done by adding a setting (if you know where to change code…) quite easily as well.
As I know the code quite well and the changes are small I’m willing to do the changes myself, I would appreciate help though. Especially in testing the changes or if you make the GUI nicer (I never do have the patience to make nice GUIs).

Thanks for coming back to me. Just to clarify my situation, I started a little company 3 years ago and wrote a linux server distro (based on Gentoo) which we now have running in about 50 SME’s all over the country (South Africa), our staff compliment is now 15 and we look after roughly roughly 2000 end users. We are a remote support company which uses the server as a drop in replacement for MS SBS and so it handles all the remote support (VNC) routing, backups (offsite via anastasi, an open source offsite backup app I wrote a while back). While we have implemented alot of open source products to provide our service, such as VNC, Cyrus Imap, OpenVPN, Rsync, Exim SMTP, Pidgin [Jabber], I haven’t been able to find an open source product that does what yours does with the ease in which it does it. I wrote a small backup app based on the mounting of samba shares then rsyncing the contents over, but this is unfortunately prone to nasty little errors, the main one being that we can’t backup a live PST file, which yours seems to do (I assume via Volume Shadowing).

Anyway, I digress, I want to roll your app out to my 50 servers/2000 end user machines to facilitate the backing up of the client workstations, and telling you my little story up there will illustrate why I need what I need, so let me clarify a little.

  1. The pc label is important because a) sometimes there are duplicates on the network, b) we name machines 1-100 internally, which allows us to vnc into them/find them on the network regardless of their name, because we allocate them an IP based on their machine number. So what I would be looking for is the ability (on the client side) to not use the pc name to identify the machine, but rather change the label to say “machine_12” so we can identify the backups more readily on the server and thus restore them. e.g. Under settings, a place we can set the label to be used (it would default to the windows machine name)

  2. The label of the backup path is important because if I add say “My Documents” to the backup path, you convert that into “My_Documents” to identify that backup path on the server. Now I see in your sqlite database that you have the path and the name in there. I have already hacked one of my test clients and changed its path name to something else (in the backupdirs table) and it just works with its new label on the server. Here it is…

    sqlite> select * from backupdirs;
    E:\My Documents|1|My_Documents
    C:\Documents and Settings\Darren|2|Darren
    C:\Lxkx75|3|mytest

You can see the last entry should be named Lxkx75, but its named mytest and so the result on the server is

root@mooplewump:/data3/BACKUP/urbackup/test/current# ls -la
total 20
drwxr-x---  5 urbackup urbackup 4096 2011-04-30 19:47 .
drwxr-x--- 14 urbackup urbackup 4096 2011-04-30 19:47 ..
drwxr-x--- 16 urbackup urbackup 4096 2011-04-30 19:47 Darren
drwxr-x---  3 urbackup urbackup 4096 2011-04-30 19:31 My_Documents
drwxr-x---  4 urbackup urbackup 4096 2011-04-30 19:47 mytest  

The issue occurs if I have both Jane and Jon logging into machine 12, now I want to backup My Documents for both of them. So one gets My_Documents, the other gets My_Documents_0. So which is Janes and which is Jons. I also see that the moment you add a path, my hack gets reset, i.e. c:\Lxkx75 reverts back to being called Lxkx75.

So my request here is if we can set this via the GUI, so when I setup Jons My Documents, I can Label it “Jons_My_Documents” etc. Is that a hard thing to do?

  1. I think you get me on this.

Regarding you being willing to make the changes, I think thats just magic thanks man. I am a very aggressive tester, because I am rolling out volume so if it were to break, I would get it in the neck from all my clients. And when it comes to GUI’s, I know next to nothing about writing windows code, but on the server side, I can get some of my staff to work on the server side GUI and also make a nice windows Tray Icon. Would that be helpful?

As an aside, there is a bug in your server, if you connect to the 55413 port with a browser, you crash the server app.

BTW, thanks for being so open in your responses, I would like to see this little app of yours get really big.

Also dude, I am going to be writing an ebuild for your server app, currently your source make file seems to only like debian based systems, so I would like to hack at it and get it to install on Gentoo seamlessly. Is that cool?

Also about the extention filtering, how hard would it be to filter based on regular expression (say PCRE) then we can skip any files/paths we don’t like the look of (like Temporary Internet Files etc).

Sure… If you have something usefull and need developer access sharing e.g. gentoo packages would be appreciated of course :)

I thought of simple bash like pattern matching (with * and ?). Is that not enough? I’m already using boost on the client so full pattern matching wouldn’t be too difficult, if perhaps less userfriendly. Is there a usecase where you need more than the glob patterns?
(Temporary internet files could be excluded by /Temporary Internet Files)

I got all your points. I’ll have time next week I think.

Re the regular expressions, you’re right, it’s overkill. Filename globbing should be fine. When it comes down to creating the gentoo package I will see if I should hack your makefile to get it to work for gentoo or I do the ‘make install’ in the ebuild itself, I will see which is more appropriate and do it that way.

Looking forward to taking it for a spin. :)

Here you go:
http://urserver.de/urbackup/unstable/

Hey man.

Anastasi is currently at internal ver 1.1 and i haven’t gotten around to making a website for it. when I complete my release of Baseline Distro 6.0 within the next month, I will include Anastasi into our freecode gentoo repository, and hopefully make it distro independant. It also needs documentation/web interface into the server as currently the config files look something like this.

# Define the job
name=universe_server_backup
description=Backup Universe /data,/chroot,/etc directories to Baseline Archive
client.email=xxxxx@example.org
client.basename=universe
office.basename=universe
debug=1  

# Server not available handler, email support
event.+.type=error
event.-.code=action_no_source_available
event.-.handler.+.class=Anastasi::EventHandler::Mail
event.-.handler.-.name=source_notify_user_handler
event.-.handler.-.config.subject=Backup failed : Source not available
event.-.handler.-.config.template=baseline_server_not_available.tmpl
event.-.handler.-.config.to.+.address=@mail.support@
event.-.handler.-.config.cc.+.address=@mail.log@
# Only if 2 consecutive errors
#event.-.handler.-.threshold.+.type=consecutive
#event.-.handler.-.threshold.-.args.threshold=2
# Only notify 1 per day
event.-.handler.-.limit.period=86400
event.-.handler.-.limit.handles=1
# Success Handler, mail backup log
event.+.type=job
event.-.code=succeed
event.-.handler.+.class=Anastasi::EventHandler::Mail
event.-.handler.-.name=notify_baseline_job_succeed
event.-.handler.-.config.subject=Backup complete - Success
event.-.handler.-.config.template=baseline_server_backup_complete_succeed.tmpl
event.-.handler.-.config.to.+.address=@client.email@
event.-.handler.-.config.cc.+.address=@mail.log@
# Failure handler, mail support
event.+.type=job
event.-.code=fail
event.-.handler.+.class=Anastasi::EventHandler::Mail
event.-.handler.-.name=notify_baseline_job_fail
event.-.handler.-.config.subject=Backup complete - Failed
event.-.handler.-.config.template=baseline_server_backup_complete_fail.tmpl
event.-.handler.-.config.to.+.address=@mail.support@
event.-.handler.-.config.to.+.address=@client.email@
event.-.handler.-.config.cc.+.address=@mail.log@
# Start Handler, mail backup log
event.+.type=job
event.-.code=begin
event.-.handler.+.class=Anastasi::EventHandler::Mail
event.-.handler.-.name=notify_user_backup_begin
event.-.handler.-.config.subject=Backup is starting
event.-.handler.-.config.template=baseline_server_backup_start.tmpl
event.-.handler.-.config.to.+.address=@client.email@
event.-.handler.-.config.cc.+.address=@mail.log@
# Complete Handler, mail backup log
event.+.type=job
event.-.code=complete
event.-.handler.+.class=Anastasi::EventHandler::Mail
event.-.handler.-.name=notify_user_job_complete
event.-.handler.-.config.subject=Backup has completed
event.-.handler.-.config.template=baseline_server_backup_complete.tmpl
event.-.handler.-.config.to.+.address=@mail.log@
# 10 hours limit
run_limit.+.type=time
run_limit.-.limit=36000
# 1.4 GB limit
run_limit.+.type=size
run_limit.-.limit=1610612736  

# Define the source
source.config.debug=0
source.class=Anastasi::Source::SSHMultiPath
source.config.hostname=@office.basename@.lclients.baselineit.net
source.config.username=root
source.config.paths.+=/data
source.config.paths.+=/etc
source.config.paths.+=/chroot
source.config.paths.+=/var  

# Define the backupset type and target
backupset.config.debug=0
backupset.class=Anastasi::BackupSet::SinglePassive
backupset.config.target.class=Anastasi::Target::Local
backupset.config.target.config.debug=0
backupset.config.target.config.path=/backups/@client.basename@/@office.basename@/  

# Define the action
action.config.debug=0
action.class=Anastasi::Action::IncrementalSync
action.config.exclude.types.music=1
action.config.exclude.types.images=1
action.config.exclude.types.video=1
action.config.exclude.types.executables=1
# exclude personal music/images
action.config.exclude.folders.+=**/RECYCLER
action.config.exclude.folders.+=data/samba/USER/*/My Documents/My Pictures
action.config.exclude.folders.+=data/samba/USER/*/My Documents/My Music
# These are operation folders, not required
action.config.exclude.folders.+=data/samba/USER/*/.spamassassin
action.config.exclude.folders.+=data/samba/USER/*/.razor
action.config.exclude.folders.+=data/mail/imap/db
action.config.exclude.folders.+=data/samba/USER/utility
# Remember to dump the ldap database
action.config.exclude.folders.+=data/mail/imap/db.backup1
action.config.exclude.folders.+=data/mail/imap/db.backup2
# Known temp files
action.config.exclude.folders.+=**/Local Settings/Temporary Internet Files
action.config.exclude.folders.+=**/Local Settings/Temp
# Don't backup imap/proc
action.config.exclude.folders.+=data/mail/imap/proc
# Don't backup useless var folders
action.config.exclude.folders.+=var/spool/exim
action.config.exclude.folders.+=var/log
action.config.exclude.folders.+=var/cache
action.config.exclude.folders.+=var/www
action.config.exclude.folders.+=var/run
# No pid files
action.config.exclude.extensions.+=pid
action.config.exclude.extensions.+=vob
action.config.exclude.empty_folders=1
action.config.compress=1
action.config.no_simulate=1  

So while its extremely flexible, its also a bitch to configure, I will tidy it up and simplify it for the next release and get a website up with docs et al.

I will have a look at that devel release now, thanks man

The code for anastasi is here
http://freecode.baselineit.net/apps/anastasi-1.1.tar.bz2
it will install on most systems, but the configuration is totally undocumented :) Give me some time to finish that :)

Hi Rendezz
very nice news see other improvements on UrBackup, is already good software but have more feature is better :-)
Perhaps what is anastasi?
Sincerely

Anastasi is an offsite backup piece of software built mostly on rsync, but the framework can handle just about anything, simply put its a framework for having a source, a target, a backup type and a backupset. But like I said its very rough, needs some polish and docs, once that is done I will release it properly and people can have a good look at it. At the moment I am caught between the release of the next version of my linux distro, running my business, incorporating urbackup into the distro and anastasi, so I don’t have as much time as I would like to work on it, but I will get there eventually.

Not yet, its working like a charm. I am currently testing the exclusion. Can I specify patterns seperated by ,?

Hey dude

Got one of my guys re-working the server side web interface (design point of view). I will let you know when its done so you can see if you like it.

Thanks

Still early stages in the design, but this is what we are looking at.

Quick Question, what IDE are you coding in, I would like to get a functional dev env up and running and it would be easier to just use the same IDE as you do.

Hi Uroni

I have had a look at some of the code and I realise my C++ is rusty as hell, can you give me some pointers as to setting up a dev environment with your structure. I would love to help with the project, but having coded Java and Perl for the last 7 years or so, I am a bit out of touch with the IDE’s and code structures of modern C++.

Thanks man