Urbackup crashing when backup goes 100%

enriquesalazar · December 17, 2013, 2:02pm

[quote=“uroni”]If they aren’t backed up it is not intentional.
Can you take a look at the C:\Program Files\UrBackup\urbackup\data\filelist.ub files to see if the files are correctly in there?
Are there any errors in the logs?
How are your include/exclude settings and the backup paths?
Which file system are you using on the server?

I just tested with 65536 files and it seemed to work. Are your files directly in a backup path or a subfolder?[/quote]

The folder is 85GB in 124.609 files.

The files seems to be correctly in filelist.ub, it has 181.886 lines.

Excluded files (with wildcards): .mp3;.avi;.mkv;.mp4;.mpg;.mpeg;*.mov
Included files (with wildcards): Users:\Documents*;Users:\Desktop*;Users:\Google Drive*
Default directories to backup: C:\Users

The folder path is C:\Users\enriquesalazar\Documents\cruce hogar, and other folders on C:\Users\enriquesalazar\Documents are being backed up without problem.

The filesystem: /dev/xvdb on /media/BACKUP/urbackup type ext3 (rw,_netdev), which actually is a virtual drive atached through Xen Server.

Cant find anything in the logs.

WHat if I delete the filelist.ub file? It will try to recreate it?

Thanks!

uroni · December 17, 2013, 9:35pm

It is recreated for each backup, so you could delete it.

The same file (minus files which could not be transfered) is stored on the server side at /var/urbackup/clientlist_${clientid}.ub .
According to wikipedia ext3 has a 31998 supdirectory limit. But you are only storing files in that folder, right?
Can you have a look at a debug level log during a backup of that client?

enriquesalazar · December 18, 2013, 3:44pm

2013-12-17 10:12:58: HT: Copying file: "/media/BACKUP/urbackup/urbackup/FA2CE31219VG/131217-0841/Users/enriquesalazar/Documents/cruce hogar/Verif Banco Agrario (liliana)/VERIF. BANCO AGRARIO.xlsx"
2013-12-17 10:12:58: GT: Loaded file "~$VERIF. BANCO AGRARIO.xlsx"
2013-12-17 10:12:58: PT: Hashing file "~$VERIF. BANCO AGRARIO.xlsx"
2013-12-17 10:12:58: No old file for "1373_Antioquia Cordoba y Sucre-2.xls"
2013-12-17 10:12:58: Loading file "1373_Antioquia Cordoba y Sucre-2.xls"
2013-12-17 10:12:59: GT: Loaded file "1373_Antioquia Cordoba y Sucre-2.xls"
2013-12-17 10:12:59: PT: Hashing file "1373_Antioquia Cordoba y Sucre-2.xls"
2013-12-17 10:12:59: No old file for "1373_Elegibles Zona Norte de Santander.xlsx"
2013-12-17 10:12:59: Loading file "1373_Elegibles Zona Norte de Santander.xlsx"
2013-12-17 10:12:59: GT: Loaded file "1373_Elegibles Zona Norte de Santander.xlsx"
2013-12-17 10:12:59: No old file for "1373_Elegibles Zona Sur de Bolivar-1.xls"
2013-12-17 10:12:59: Loading file "1373_Elegibles Zona Sur de Bolivar-1.xls"
2013-12-17 10:12:59: PT: Hashing file "1373_Elegibles Zona Norte de Santander.xlsx"
2013-12-17 10:12:59: GT: Loaded file "1373_Elegibles Zona Sur de Bolivar-1.xls"
2013-12-17 10:12:59: PT: Hashing file "1373_Elegibles Zona Sur de Bolivar-1.xls"
2013-12-17 10:12:59: No old file for "1373_Solicitud de Elegibles por Territorio -BOYACÁ.xlsx"
2013-12-17 10:12:59: Loading file "1373_Solicitud de Elegibles por Territorio -BOYACÁ.xlsx"
2013-12-17 10:12:59: GT: Loaded file "1373_Solicitud de Elegibles por Territorio -BOYACÁ.xlsx"
2013-12-17 10:12:59: PT: Hashing file "1373_Solicitud de Elegibles por Territorio -BOYACÁ.xlsx"
2013-12-17 10:12:59: No old file for "1373_Solicitud de Elegibles por Territorio -CUNDINAMARCA.xlsx"
2013-12-17 10:12:59: Loading file "1373_Solicitud de Elegibles por Territorio -CUNDINAMARCA.xlsx"
2013-12-17 10:13:00: GT: Loaded file "1373_Solicitud de Elegibles por Territorio -CUNDINAMARCA.xlsx"
2013-12-17 10:13:00: PT: Hashing file "1373_Solicitud de Elegibles por Territorio -CUNDINAMARCA.xlsx"
2013-12-17 10:13:01: HT: Linked file: "/media/BACKUP/urbackup/urbackup/FA2CE31219VG/131217-0841/Users/enriquesalazar/Documents/cruce hogar/Verif Banco Agrario (liliana)/~$VERIF. BANCO AGRARIO.xlsx"
2013-12-17 10:13:01: HT: Copying file: "/media/BACKUP/urbackup/urbackup/FA2CE31219VG/131217-0841/Users/enriquesalazar/Documents/cruce hogar/Verif Banco Agrario (liliana)/CONSOLIDADO ELEGIBLES POR TERRITORIO/1373_Antioquia Cordoba y Sucre-2.xls"
2013-12-17 10:13:01: HT: Linked file: "/media/BACKUP/urbackup/urbackup/FA2CE31219VG/131217-0841/Users/enriquesalazar/Documents/cruce hogar/Verif Banco Agrario (liliana)/CONSOLIDADO ELEGIBLES POR TERRITORIO/1373_Elegibles Zona Norte de Santander.xlsx"
2013-12-17 10:13:01: HT: Linked file: "/media/BACKUP/urbackup/urbackup/FA2CE31219VG/131217-0841/Users/enriquesalazar/Documents/cruce hogar/Verif Banco Agrario (liliana)/CONSOLIDADO ELEGIBLES POR TERRITORIO/1373_Elegibles Zona Sur de Bolivar-1.xls"
2013-12-17 10:13:01: HT: Linked file: "/media/BACKUP/urbackup/urbackup/FA2CE31219VG/131217-0841/Users/enriquesalazar/Documents/cruce hogar/Verif Banco Agrario (liliana)/CONSOLIDADO ELEGIBLES POR TERRITORIO/1373_Solicitud de Elegibles por Territorio -BOYACÁ.xlsx"
2013-12-17 10:13:01: HT: Linked file: "/media/BACKUP/urbackup/urbackup/FA2CE31219VG/131217-0841/Users/enriquesalazar/Documents/cruce hogar/Verif Banco Agrario (liliana)/CONSOLIDADO ELEGIBLES POR TERRITORIO/1373_Solicitud de Elegibles por Territorio -CUNDINAMARCA.xlsx"
2013-12-17 10:13:08: Loading file patch for "Comparacion GAE - AWS 2013 v2.pptx"
2013-12-17 10:13:08: Old filesize=57627
2013-12-17 10:13:08: Receiving filesize...

Hi.

It seems the folder is being backed up. But not showing in the Backups tab in the web UI.

How else could I check?

THanks.

uroni · December 18, 2013, 9:11pm

You could go to /media/BACKUP/urbackup/urbackup/FA2CE31219VG/131217-0841/Users/enriquesalazar/Documents/cruce hogar with any file manager or with a shell.

If possible, can you check for any JavaScript errors when displaying the web page, or for what the server sends when you click on “Documents”?

enriquesalazar · December 19, 2013, 3:37pm

Hi.

Last night the server got finally full.Its been between 97 and 99% the whole past week

This is a 2TB disk, and I set the Global soft filesystem quota to 1500G.

In the log I found

2013-12-19 03:00:52: Enforcing quota for client "FA2CE31219VW" (id=95)
2013-12-19 03:00:52: Cleaning up 100 percent
2013-12-19 03:00:52: Quota enforcement report for client "FA2CE31219VW" (id=95)
Client uses 1.05172 TB and has a quota of 1.82482 TB
Client within assigned quota.  

2013-12-19 02:50:51: Enforcing quota for client "FAMXL3101S5R" (id=83)
2013-12-19 02:50:51: Cleaning up 100 percent
2013-12-19 02:51:29: Lost channel connection to client. has_error=false
2013-12-19 02:51:30: Resetting channel because of timeout.
2013-12-19 03:00:52: WARNING: Sending mail failed. ec=67 -- Access denied: 451
2013-12-19 03:00:52: ERROR: Quota enforcement report for client "FAMXL3101S5R" (id=83)
Client uses 13.7902 TB and has a quota of 1.82482 TB
This requires enforcement of the quota.
Error. Target space is negative

Maybe Urbackup is not cleaning the files from the previous install (before I started using the nighties)? or maybe when the database got corrupted?

enriquesalazar · December 24, 2013, 1:01pm

Hi…After a long while, the server crashed again. here the gdb output:

2013-12-24 06:42:40: Socket has error: 113
2013-12-24 06:42:40: Connecting Channel to ClientService failed - CONNECT error -55
2013-12-24 06:42:53: Socket has error: 113
2013-12-24 06:42:53: Connecting Channel to ClientService failed - CONNECT error -55
2013-12-24 06:43:06: Socket has error: 113
2013-12-24 06:43:06: Connecting Channel to ClientService failed - CONNECT error -55  

Program received signal SIGHUP, Hangup.
0xb7fdd424 in __kernel_vsyscall ()
(gdb) bt
#0  0xb7fdd424 in __kernel_vsyscall ()
#1  0xb7de2460 in poll () from /lib/i386-linux-gnu/libc.so.6
#2  0x08052179 in CAcceptThread::operator() (this=0x8162860, single=true)
    at AcceptThread.cpp:153
#3  0x0807a470 in main_fkt (argc=31, argv=0xbffff554) at main.cpp:460
#4  0x08050ceb in main (argc=31, argv=0xbffff554) at main.cpp:106
(gdb)

uroni · December 27, 2013, 11:46am

Thats expected. That is the logrotation. Enter “handle SIGHUP nostop” to ignore that signal. (Or enter “continue” if you get this again).

enriquesalazar · December 27, 2013, 1:12pm

Ok thanks.

I still have one folder with like 50 1GB+ files, and the server is not backing it. Any way to troubleshoot it?

Thanks!

uroni · December 27, 2013, 1:57pm

Hmm, last time you said it gets backed up but not shown on the web interface. Have you confirmed that? (By navigating to the storage path on the server)

enriquesalazar · December 27, 2013, 3:37pm

My bad. in the library windows mixes public documents with my documents, I was looking for the folder in my documents, when it was in public documents the whole time.

Where can I see the size of a client’s full backup? STATISTICS shows an error “Negative values are invalid for a pie chart.”

Thanks again!

enriquesalazar · December 27, 2013, 4:37pm

I also had this

#0  0xb7fdd424 in __kernel_vsyscall ()
#1  0xb7d2f1df in raise () from /lib/i386-linux-gnu/libc.so.6
#2  0xb7d32825 in abort () from /lib/i386-linux-gnu/libc.so.6
#3  0xb7f9213d in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/i386-linux-gnu/libstdc++.so.6
#4  0xb7f8fed3 in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#5  0xb7f8ff0f in std::terminate() ()
   from /usr/lib/i386-linux-gnu/libstdc++.so.6
#6  0xb7f9005e in __cxa_throw () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#7  0xb7f34113 in std::__throw_length_error(char const*) ()
   from /usr/lib/i386-linux-gnu/libstdc++.so.6
#8  0xb7f76e65 in std::string::resize(unsigned int, char) ()
   from /usr/lib/i386-linux-gnu/libstdc++.so.6
#9  0x08092be3 in resize (__n=1073741974, this=0xb38de1ac)
    at /usr/include/c++/4.6/bits/basic_string.h:749
#10 CStringOutputStream::write (this=0xb38de1a8, 
    buf=0xb38dd0ac "&parentesco=Jefe%28a%29+o+cabeza+del+hogar&ostenta_tenencia=0&tipo_tenencia=Propio+totalmente+pagado&estado_civil=Divorciado&edad=47&sexo=Masculino&ocupacion=Independiente&ingresos_mensuales=300000&po"..., count=4096, 
    stream=STDOUT) at OutputStream.cpp:38
#11 0x08064cc9 in CServer::WriteRaw (this=0x0, tid=40, 
    buf=0xb38dd0ac "&parentesco=Jefe%28a%29+o+cabeza+del+hogar&ostenta_tenencia=0&tipo_tenencia=Propio+totalmente+pagado&estado_civil=Divorciado&edad=47&sexo=Masculino&ocupacion=Independiente&ingresos_mensuales=300000&po"..., bsize=4096, 
    cached=false) at Server.cpp:625
#12 0xb6eb6082 in Actions::backups::Execute (this=0x815cd88, GET=..., 
    POST=..., tid=40, PARAMS=...) at serverinterface/backups.cpp:209
#13 0x08065755 in CServer::Execute (this=0x81332e0, action=..., context=..., 
    GET=..., POST=..., PARAMS=..., req=0xb38de1a8) at Server.cpp:490
#14 0x0805f055 in CServer::Execute (this=0x0, action=..., context=..., 
    GET=..., POST=..., PARAMS=...) at Server.cpp:510
#15 0xb7070c36 in CHTTPAction::operator() (this=0xb2a14d58)
    at HTTPAction.cpp:50
#16 0x08093742 in CPoolThread::operator() (this=0xb6911b48)
    at ThreadPool.cpp:42
#17 0x0805e42e in thread_helper_f (t=0xb6911b48) at Server.cpp:1203
#18 0xb7eb1d4c in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#19 0xb7df0bae in clone () from /lib/i386-linux-gnu/libc.so.6

I guess is not the same error

uroni · December 29, 2013, 3:17pm

Yes. This seems to be a big file downloaded via the integrated http server. The file is loaded completely into memory before being sent. I’m fixing this but you can also use a “real” web server like apache in front of UrBackup. See http://www.urbackup.org/Administration_Manual.php#x1-80003.2

felipep · January 9, 2014, 3:28pm

I am getting the same behavior… crashes when it hits 100%… every time… I am running the Windows version… a minidump is created and when I loaded in VS…
Summary:
Exception Code 0xC0000005
Exception Information The thread tried to read from or write to a virtual address for which it does not have the appropriate access

CallStack:
msvcr100.dll!memcpy()
urbackup_srv.exe!00000001400019d4()
urbackup_srv.exe!00000001400076d7()
…
msvcr100.dll!endthreadex()
msvcr100.dll!endthreadex()

Disassembly of the memcpy() around the crash:
0000000077ADC1F9 77 B5 ja memcpy+250h (077ADC1B0h)
0000000077ADC1FB B8 20 00 00 00 mov eax,20h
0000000077ADC200 48 81 E9 80 00 00 00 sub rcx,80h
0000000077ADC207 0F 18 04 0A prefetchnta [rdx+rcx]
0000000077ADC20B 0F 18 44 0A 40 prefetchnta [rdx+rcx+40h]
0000000077ADC210 FF C8 dec eax
0000000077ADC212 75 EC jne memcpy+2A0h (077ADC200h)
0000000077ADC214 48 81 C1 00 10 00 00 add rcx,1000h
0000000077ADC21B B8 40 00 00 00 mov eax,40h
***0000000077ADC220 4C 8B 4C 0A F8 mov r9,qword ptr [rdx+rcx-8]

Disassembly of the calling urbackup_srv.exe code:
0000000140001900 48 89 5C 24 08 mov qword ptr [rsp+8],rbx
0000000140001905 48 89 6C 24 10 mov qword ptr [rsp+10h],rbp
000000014000190A 48 89 74 24 18 mov qword ptr [rsp+18h],rsi
000000014000190F 57 push rdi
0000000140001910 48 83 EC 20 sub rsp,20h
0000000140001914 48 8B 7A 10 mov rdi,qword ptr [rdx+10h]
0000000140001918 49 8B E8 mov rbp,r8
000000014000191B 48 8B F2 mov rsi,rdx
000000014000191E 48 8B D9 mov rbx,rcx
0000000140001921 49 3B F8 cmp rdi,r8
0000000140001924 73 0E jae 0000000140001934
0000000140001926 48 8D 0D 1B 67 0C 00 lea rcx,[1400C8048h]
000000014000192D FF 15 C5 5B 0C 00 call qword ptr [1400C74F8h]
0000000140001933 CC int 3
0000000140001934 49 2B F8 sub rdi,r8
0000000140001937 4C 3B CF cmp r9,rdi
000000014000193A 49 0F 42 F9 cmovb rdi,r9
000000014000193E 48 3B CA cmp rcx,rdx
0000000140001941 75 1F jne 0000000140001962
0000000140001943 4A 8D 14 07 lea rdx,[rdi+r8]
0000000140001947 49 83 C8 FF or r8,0FFFFFFFFFFFFFFFFh
000000014000194B E8 C0 00 00 00 call 0000000140001A10
0000000140001950 4C 8B C5 mov r8,rbp
0000000140001953 33 D2 xor edx,edx
0000000140001955 48 8B CB mov rcx,rbx
0000000140001958 E8 B3 00 00 00 call 0000000140001A10
000000014000195D E9 89 00 00 00 jmp 00000001400019EB
0000000140001962 48 83 FF FE cmp rdi,0FFFFFFFFFFFFFFFEh
0000000140001966 76 0E jbe 0000000140001976
0000000140001968 48 8D 0D C9 66 0C 00 lea rcx,[1400C8038h]
000000014000196F FF 15 7B 5B 0C 00 call qword ptr [1400C74F0h]
0000000140001975 CC int 3
0000000140001976 48 8B 41 18 mov rax,qword ptr [rcx+18h]
000000014000197A 48 3B C7 cmp rax,rdi
000000014000197D 73 27 jae 00000001400019A6
000000014000197F 4C 8B 41 10 mov r8,qword ptr [rcx+10h]
0000000140001983 48 8B D7 mov rdx,rdi
0000000140001986 E8 25 03 00 00 call 0000000140001CB0
000000014000198B 48 85 FF test rdi,rdi
000000014000198E 74 5B je 00000001400019EB
0000000140001990 48 83 7E 18 10 cmp qword ptr [rsi+18h],10h
0000000140001995 72 03 jb 000000014000199A
0000000140001997 48 8B 36 mov rsi,qword ptr [rsi]
000000014000199A 48 83 7B 18 10 cmp qword ptr [rbx+18h],10h
000000014000199F 72 24 jb 00000001400019C5
00000001400019A1 48 8B 0B mov rcx,qword ptr [rbx]
00000001400019A4 EB 22 jmp 00000001400019C8
00000001400019A6 48 85 FF test rdi,rdi
00000001400019A9 75 E5 jne 0000000140001990
00000001400019AB 48 89 79 10 mov qword ptr [rcx+10h],rdi
00000001400019AF 48 83 F8 10 cmp rax,10h
00000001400019B3 72 08 jb 00000001400019BD
00000001400019B5 48 8B 01 mov rax,qword ptr [rcx]
00000001400019B8 40 88 38 mov byte ptr [rax],dil
00000001400019BB EB 2E jmp 00000001400019EB
00000001400019BD 48 8B C1 mov rax,rcx
00000001400019C0 C6 01 00 mov byte ptr [rcx],0
00000001400019C3 EB 26 jmp 00000001400019EB
00000001400019C5 48 8B CB mov rcx,rbx
00000001400019C8 48 8D 14 2E lea rdx,[rsi+rbp]
00000001400019CC 4C 8B C7 mov r8,rdi
00000001400019CF E8 18 F5 0B 00 call 00000001400C0EEC
00000001400019D4 48 83 7B 18 10 cmp qword ptr [rbx+18h],10h

felipep · January 9, 2014, 3:39pm

I pressed sent too quickly… the web log looks almost identical every time:
09.01.14 10:06 INFO server_prepare_hash Thread finished (exit)
09.01.14 10:06 DEBUG Setting cachesize to 40960
09.01.14 10:06 INFO server_hash Thread finished - normal
09.01.14 10:06 ERROR Fatal exception (APPLICATION CRASHED). Crash dump written to “C:\Windows\TEMP\UrBackup\v0.25.1-20140109-100627-10532-12000.dmp”

uroni · January 9, 2014, 9:00pm

It seems to be at about the same place where it crashed with the SQLite file entry cache with 1.3.1 (fixed in 1.3.2).

Could you send me one of the minidumps (to martin@urbackup.org)?

felipep · January 9, 2014, 9:20pm

You are correct, I set “Cache database type for file entries” to None and the error went away. Do you still want me to send you the minidump?

uroni · January 9, 2014, 10:21pm

If you are using 1.3.2, that would be nice.

zidvbelju · January 23, 2014, 3:50pm

EDIT- Posted this in the wrong thread. Disregard.