Async indexing process not found

[pjablonski]

We are seeing this error regularly with most of our Windows machines. Running client 2.1.15 with server 2.1.19 (Centos 6.x).

2017-05-17 04:12:08(info): Starting scheduled full file backup…
2017-05-17 04:21:15(error): Constructing of filelist of “HOOVER” failed: error - Async indexing process not found
2017-05-17 04:21:25(error): Backup had an early error. Deleting partial backup.

A review of the client debug.log shows:

2017-05-12 09:29:28: WARNING: Info not found at ‘c:’ - reindexing
2017-05-12 09:29:37: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-05-12 09:29:37: ERROR: Last error: 10058
2017-05-12 09:30:28: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-05-12 09:30:28: ERROR: Last error: 10058
2017-05-12 09:32:10: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-05-12 09:32:10: ERROR: Last error: 10058
2017-05-12 09:37:16: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-05-12 09:37:16: ERROR: Last error: 10058
2017-05-12 10:01:13: WARNING: Limiting number of DC users to 102
2017-05-12 10:04:18: WARNING: Limiting number of DC groups to 102
2017-05-12 10:04:20: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:31: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:35: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:37: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:38: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:40: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:41: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:43: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:44: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:45: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:52: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:53: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:58: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:04:59: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:01: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:02: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:03: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:05: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:17: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:18: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:20: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:21: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:23: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:24: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:25: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:27: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:28: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:30: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:31: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:36: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:40: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:42: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:47: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:48: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:50: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:55: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:56: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:05:58: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:04: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:05: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:07: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:08: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:16: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:17: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:22: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:24: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:25: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:33: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:34: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:36: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:37: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:39: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:40: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:42: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:43: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:45: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:51: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:53: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:54: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:55: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:57: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:06:58: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:07:08: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:07:11: ERROR: SQLITE: Long running query Stmt: [BEGIN IMMEDIATE;]
2017-05-12 10:07:11: ERROR: Active query(0): BEGIN IMMEDIATE;
2017-05-12 10:07:15: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:07:20: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:07:21: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:07:29: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:07:34: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:07:36: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:07:41: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:07:42: ERROR: Error while enumerating DC user groups: 234
2017-05-12 10:08:46: ERROR: Fatal exception code 3221225725 at address 0x00000000771ADD43
2017-05-12 10:48:06: WARNING: There are 5584416 new USN entries at ‘c:’ - reindexing
2017-05-12 10:49:27: WARNING: Indexing was not finished at ‘c:’ - reindexing
2017-05-12 10:50:58: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-05-12 10:50:58: ERROR: Last error: 10058
2017-05-12 11:30:14: WARNING: Limiting number of DC users to 102
2017-05-12 11:33:17: WARNING: Limiting number of DC groups to 102
2017-05-12 11:34:09: ERROR: Fatal exception code 3221225725 at address 0x0000000076F1DD43
2017-05-16 04:56:07: WARNING: Last record not readable at ‘c:’ - reindexing. Last record USN is 54490230064 FirstUsn is 55097688064 NextUsn is 55133845224
2017-05-16 04:56:07: WARNING: There are 643615160 new USN entries at ‘c:’ - reindexing
2017-05-16 04:57:06: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-05-16 04:57:06: ERROR: Last error: 10058
2017-05-16 05:02:22: WARNING: Limiting number of DC users to 102
2017-05-16 05:05:27: WARNING: Limiting number of DC groups to 102
2017-05-16 05:06:20: ERROR: Fatal exception code 3221225725 at address 0x0000000076F1DD43
2017-05-16 05:33:06: WARNING: There are 2732784 new USN entries at ‘c:’ - reindexing
2017-05-16 05:34:22: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-05-16 05:34:22: ERROR: Last error: 10058
2017-05-16 05:41:12: WARNING: Limiting number of DC users to 102
2017-05-16 05:44:13: WARNING: Limiting number of DC groups to 102
2017-05-16 05:45:07: ERROR: Fatal exception code 3221225725 at address 0x0000000076F1DD43
2017-05-16 06:30:36: WARNING: Limiting number of DC users to 102
2017-05-16 06:33:37: WARNING: Limiting number of DC groups to 102
2017-05-16 06:34:27: ERROR: Fatal exception code 3221225725 at address 0x0000000076F1DD43
2017-05-16 08:00:21: WARNING: Limiting number of DC users to 102
2017-05-16 08:03:22: WARNING: Limiting number of DC groups to 102
2017-05-16 08:04:10: ERROR: Fatal exception code 3221225725 at address 0x0000000076F1DD43
2017-05-16 11:54:43: WARNING: Limiting number of DC users to 102
2017-05-16 11:57:44: WARNING: Limiting number of DC groups to 102
2017-05-16 11:58:32: ERROR: Fatal exception code 3221225725 at address 0x0000000076F1DD43
2017-05-16 12:07:45: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-05-16 12:07:45: ERROR: Last error: 0
2017-05-16 17:26:58: WARNING: Limiting number of DC users to 102
2017-05-16 17:30:04: WARNING: Limiting number of DC groups to 102
2017-05-16 17:30:53: ERROR: Fatal exception code 3221225725 at address 0x0000000076F1DD43
2017-05-17 04:16:21: WARNING: Limiting number of DC users to 102
2017-05-17 04:19:24: WARNING: Limiting number of DC groups to 102
2017-05-17 04:20:12: ERROR: Fatal exception code 3221225725 at address 0x0000000076F1DD43
2017-05-17 04:21:15: WARNING: There are 1785528 new USN entries at ‘c:’ - reindexing

Yes, these machines are on an Active Directory (less than 500 clients). I believe this is related to an earlier post:

Connecting for filelist and nothing more

You are not this guy who want’s to create one AD user for every Australian https://www.reddit.com/r/sysadmin/comments/5m293q/active_directory_for_28_million_users/ Just joking. But how many AD users do you have? Seems this needs to be optimized/rewritten with more AD users in mind.

In addition the backup speed has been horrendous (KB/s) since we upgraded to the 2.x client and 2.x server. That’s probably a separate question, but we would love to solve the async errors we’re seeing as it appears many machines are not getting backed up.

Backup times are in years in many cases:

Advice?

[pjablonski]

After spending several hours trying to troubleshoot this I am no further along than when originally reported. In an effort to determine if this is a client issue, I have removed the 2.1.15 client completely (used Windows uninstall and deleted any files in C:\program files\UrBackup) and re-installed 2.0.36. This machine, hoover, is still dying with Timeout(1) and now the client log looks like this (only last few lines shown):

2017-06-05 09:33:47: WARNING: Parent directory not found in MFT. Was probably deleted.
2017-06-05 09:33:47: WARNING: Parent of file with FRN 88664617664112585 (Name “f_01bf1b”) with FRN 62487444830343005 not found. Searching via MFT as fallback.
2017-06-05 09:33:47: WARNING: Parent directory not found in MFT. Was probably deleted.
2017-06-05 09:33:47: WARNING: Parent of file with FRN 14355223812898912 (Name “index”) with FRN 62487444830343005 not found. Searching via MFT as fallback.
2017-06-05 09:33:47: WARNING: Parent directory not found in MFT. Was probably deleted.
2017-06-05 09:33:47: WARNING: Parent of file with FRN 162974011515514436 (Name “Report.wer”) with FRN 398568567022333183 not found. Searching via MFT as fallback.
2017-06-05 09:33:47: WARNING: Parent of file with FRN 163255486492225092 (Name “bos_Subordinate_Cert.cer”) with FRN 11258999068474411 not found. Searching via MFT as fallback.
2017-06-05 09:34:09: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-06-05 09:34:09: ERROR: Last error: 10058
2017-06-05 09:37:10: ERROR: Fatal exception code 3221225725 at address 0x00000000772EC8CC
2017-06-06 07:03:02: WARNING: Last record not readable at ‘c:’ - reindexing. Last record USN is 58632081336 FirstUsn is 58763509760 NextUsn is 58800444136
2017-06-06 07:03:02: WARNING: There are 168362800 new USN entries at ‘c:’ - reindexing
2017-06-06 07:04:04: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-06-06 07:04:04: ERROR: Last error: 10058
2017-06-06 07:06:50: ERROR: Fatal exception code 3221225725 at address 0x00000000772EC8CC
2017-06-06 07:14:59: ERROR: Recvfrom error in CUDPThread::UdpStep
2017-06-06 07:14:59: ERROR: Last error: 0
2017-06-06 07:15:52: ERROR: Fatal exception code 3221225725 at address 0x00000000772EC8CC

I must not be the only one having these issues. Anybody have any further troubleshooting advice? Thanks.

[uroni]

@pjablonski

2017-06-06 07:15:52: ERROR: Fatal exception code 3221225725 at address 0x00000000772EC8CC

Those are crashes. Unforunately it does not create a crashdump. If you manage to create crashdumps I could have a look at them…

[pjablonski]

Thanks uroni. I assume we just wait for a crashdump? Right now all the crashdump files are empty.

[uroni]

Perhaps one of those methods works: https://blogs.msdn.microsoft.com/cobold/2010/03/01/collecting-crash-dumps/

[pjablonski]

I’ve managed to capture a few crashdumps. Let me know where I can send. Thanks.

[uroni]

See here: Having problems with UrBackup? Please read before posting Thanks!

[pjablonski]

Thanks. Dumps just sent as described. Should arrive shortly.

[pjablonski]

The zipped crashdump and client log are 24 MB and email fails. Is there a upload site where I can place? Thanks.

[uroni]

Looks like there is some kind of loop at C:\Users\maXXXXXX\Documents\TIJ\out\test\TIJ\test\TIJ\test\TIJ

Which client version is this?

[pjablonski]

Wrong response for this post?

[uroni]

No

[pjablonski]

If not, then we’re using 2.0.36 as it avoids the async timeouts (at least in some cases). I will exclude: C:\Users\maXXXXXX\Documents\TIJ* and test again.

[pjablonski]

Exclusion resulted in same timeout.

06/07/17 08:12 INFO Starting unscheduled full file backup…
06/07/17 08:12 DEBUG HOOVER: Connecting for filelist…
06/07/17 08:12 DEBUG HOOVER: Waiting for filelist
06/07/17 08:13 ERROR Constructing of filelist of “HOOVER” failed - TIMEOUT(1)
06/07/17 08:13 ERROR Backup had an early error. Deleting partial backup.

[uroni]

Hmm, looks like a normal, but deep folder structure. I’ll run some tests.

[pjablonski]

Thank you.

[pjablonski]

Hi Uroni. Have your tests revealed anything?

[uroni]

Yeah, increasing the stack size for the indexing thread fixes it.

[pjablonski]

Thanks. What’s the recommended procedure for increasing the stack size or is that done in a patch to the client.

[pjablonski]

Any thoughts on this? Thanks.