View Issue Details

IDProjectCategoryView StatusLast Update
0002615MMW v4Synchronizationpublic2012-04-30 13:41
Reporterrusty Assigned To 
PriorityurgentSeveritymajorReproducibilityhave not tried
Status closedResolutionfixed 
Target Version4.0Fixed in Version4.0 
Summary0002615: MTP Synchronization is MUCH slower than it should be
DescriptionI'm not sure why this would be the case, but there are 2 cases that seem to dramatically slow down synchronization:

1) The existence of many Dead Links. I'm not sure why this would be the case, considering that 'Dead Links' are supposed to be ignored in the synch dialog), but 2 users have indicated that by deleting files in the 'dead links' node, they've dramatically speeded up synchronization with MTP devices.

2) The existence of many other files on a device. e.g. I tried to auto-sync a single Album that contained a single track to a USB Key, and MediaMonkey was 'Preparing list of tracks' for > 5 minutes. Upon looking at the debug log, it seems that MM is enumerating the entire contents of the USB key instead of just looking at the directory to which tracks are being synched.

e.g.
USB key has several directories:
/Apps
/Documents
/Temp

MM is configured to sync Album="A Quick one" to:
/My Music/<Artist>/<Album>/<Artist>-<Title>

What I would expect is that MM should examine the contents of the /My Music directory on the device to the sync list. Since there is nothing in the /My Music directory (it doesn't exist), then MM should just copy over the single file. Instead, MM seems to examine a large portion of the 4GB USB key!!

Note: This issue may apply to other sync plugins--haven't tested this. Also note: the debug log shows this happening with an MPC file (i.e. auto-conversion requiered) but the bug also occurs with MP3 files.
Additional InformationCase 1 Reported at: http://www.mediamonkey.com/forum/viewtopic.php?t=10834
Debug log for case 2 posted to the ftp server

Case 2) Reported at:
http://www.mediamonkey.com/forum/viewtopic.php?f=6&t=38674 [^]
http://www.mediamonkey.com/support/staff/index.php?_m=tickets&_a=viewticket&ticketid=4417 [^]
http://www.mediamonkey.com/support/staff/index.php?_m=tickets&_a=viewticket&ticketid=4787 [^]
TagsNo tags attached.
Fixed in build1344

Relationships

related to 0001350 closedrusty Portable Device: Bidirectional Synchronization 
related to 0005510 closedLudek Auto-sync a full WMDM device is deferred until whole device content is read 
related to 0002009 resolved MTP Device Export: performance is very slow 
related to 0003210 closedLudek Audio files are deleted when they shouldn't be deleted 
related to 0006491 closedLudek MTP Devices should use root folder for in-place configuration 
related to 0009284 closedLudek MTP performance can be very slow in some cases 

Activities

rusty

2007-06-28 17:21

administrator   ~0009615

Note that case 2) does not occur when using the Generic USB MS plugin.

jiri

2007-06-29 13:34

administrator   ~0009618

1) Should be just a coincidence, it really shouldn't have any influence.

2) The problem is that we currently need to fully scan the device, because we at least need to know which files should we delete, what to bi-synchronize, etc. We could work without that start-up scan, but then:

2a) MM couldn't delete files not on auto-synch list (or to be exact, MM could delete it knows about from the last synchronization, it couldn't delete any new files created e.g. by other applications).

2b) MM couldn't bi-synchronize, because it wouldn't know about new or changed content.

2c) This is probably the most serious problem: MM couldn't synchronize ratings and playcounts from the device back to PC (again, because MM finds about them only by scanning each track on the devices).

Btw, it's only a problem of WMDM devices, e.g. iPod doesn't have such a problem, because it has one global DB file and all the necessary information is there. (Well, actually even iPod is scanned for new files then, but it's much faster using standard filesystem operations)

rusty

2007-06-29 19:37

administrator   ~0009620

2a) If MM is configured to sync to a specific directory, i think it goes without saying that files should be deleted only if they're contained within that specific directory. Whatever is on the rest of the device is normally irrelevant. If you disagree with this, then we could add an option to:
 [ ] Delete Tracks that aren't on the Auto-sync list from the device
. . . [ ] From entire device (slow)
 Ignore these folders ______________________________________

Note: If 'From entire device' isn't enabled, then tracks are deleted only from 'hardcoded' folders in the 'Destination' (including custom destinations).

2b) Hmm... you're right. That would imply that the above approach wouldn't work, and that a better way of doing this would be to let the user define what folders on the device to bi directionally synchronize.

2c) This isn't that much of a problem as far as this bug is concerned. i.e. The slowness described in this bug occurs _even when there are no audio tracks on the device_.

Given all of the above, a possible approach might be to have the user configure which folders on the device should be deleted (and synced when we add bidi synchronization). i.e. rather than having a list of 'Exception folders', there should be a list of folders to synch, which would include by default, all 'hardcoded' Destination folders.

e.g.
 [ ] Delete Tracks that aren't on the Auto-sync list from the device
. . . From the following folders ______________________________________

Note: ideally there'd be a browser checklist to help select these folders.

Note2: When we add bidi synchronization (bug 0001350), we'll probably want to add an 'Auto-sync list Device' entry that would give the user the ability to configure how auto-synch is handled on the device.

Thoughts?

jiri

2007-06-30 11:13

administrator   ~0009627

Well, problems of the proposed solution are:
A. It would require user to manually set the value and therefore also to understand what is he/she doing.
B. I'm not sure, how much would it solve. It would certainly get rid of unnecessary scanning of folders with photos, etc., but e.g. on devices with big HDs one can still have 10k of tracks and their scanning would still take a long time.

However, I don't know if there's any better solution. One idea would be to implement some 'Quick Auto-synch' mode where the full device scan wouldn't be done and thus also some features would be disabled (bi-di synch of tracks, back-synch of ratings and playcount). I guess that it would be useful for some users. Or maybe a mixed mode where user would specify to use the proposed Quick mode, but e.g. once a week also use the Full mode - so that ratings are properly synched back. Just some thoughts..

rusty

2007-07-05 20:15

administrator   ~0009653

Last edited: 2007-07-05 20:19

I think I have a solution that meets the short term need to fix slowness of MTP devices, AND fits in with the longer term need of bidirectional synchronization:

In the short term:
 [ ] Delete Tracks that aren't on the Auto-sync list from the device
     [ ] From entire device (slow)
     Ignore these folders ______________________________________

OR

 [ ] Delete Tracks that aren't on the Auto-sync list
     from ____________________ (Auto-synced folders on the device, the entire device (slow) )
     Ignore these folders ______________________________________
The objection raised previously to this is that it:
-it's not compatible with bi-di synchronization
-it's a problem for bi-di ratings sync
The first issue isn't a problem if we take the approach outlined below for bidi synchronization. And the second issue isn't really an issue since ratings synchronization is always from tracks that fall within the 'hard' directories in the destination synch path.

In terms of the future, when we add bi-di synchronization, the UI could evolve as follows:
Auto-Sync Settings Confirm

[x] Sync new/updated tracks on the Auto-sync list to the device
    [x] Delete tracks that aren't on the Auto-sync list form the device [x]
        from ________________________^ ([auto-synched folders on the device], entire device (slow) )
    Ignore these folders: _______________________________________

[ ] Sync new/updated tracks from __________________________^ (Auto-synced folders on the device to the PC, entire device to the PC (slow) )
    [ ] Delete tracks not on the device from the PC [x]
        Save new tracks to: _______________________________________ [browse]
The only thing that needs to be fine tuned is the fact that when we add bidi support, the device directory configuration is spread across multiple tabs.

Ludek

2010-01-22 10:33

developer   ~0019970

Last edited: 2010-01-22 10:46

As discussed with Jiri over IM, this issue should be fixed before 4.0 release, because capacity of portable devices grows and most of users has more and more files on the device (e.g. all their photos etc.) Preferred approach could be:

[ ] Delete Tracks that aren't on the Auto-sync list

     from ____________________ (Auto-synced folders on the device, the entire device (slow) )

     Ignore these folders ______________________________________


In addition (Taken from 0005510: Auto-sync a full WMDM device is deferred until whole device content is read):
Progress bar should indicate what it is really doing, i.e. instead of "Preparing list of tracks" there should be something like "Reading device content" or "Synchronizing tracks info from device back to PC"

rusty

2010-01-22 14:30

administrator   ~0019972

Upon further reflection, I think that the UI might be too complicated--it assumes that the user knows which root folders tracks are being synced to.

Can't all of this be done transparently to the user (without any UI)? i.e. The synch profile contains the 'Destination' folder where tracks are synced to and/or wmdm determines where playlists are synced to. If MM 'knows' both of those bits of information, can't it determine what has changed? (i.e. what I'm saying is that it's ok if MM does bidirectional synchronization _only_ between the folders that it knows about).

Ludek

2010-01-23 00:46

developer   ~0019973

Yes, I agree. This was also my point initially and I also think that no UI change is required.

Ludek

2010-01-24 23:24

developer   ~0019979

Fixed in build 1301 as suggested.

i.e. If device is configured as
Sync Tracks to: [\Music\<Album Artist>\<Album>\<Title>]
then MM looks up only \Music\ folder and subfolders.

jiri

2010-04-02 17:00

administrator   ~0020079

Several issues:
1. Mainly, it doesn't work when mask is like '\f1\f2\<Artist>\...' i.e. with two nested folders.
2. It should take also Playlist folder into account.
3. It should be faster, taking in memory data in account, i.e. don't access DB at all, get in-memory device structure and work with it.

Ludek

2010-04-06 13:17

developer   ~0020087

Fixed in build 1305.

Ludek

2010-10-11 13:48

developer   ~0020724

Last edited: 2010-10-11 13:49

The device scanning process was significantly sped up in build 1316, also fixed some termination issues and adjusted to comply with 0006491 , i.e. now the whole device is looked up (reverted changes from builds 1301 - 1305), but the scan starts immediatelly after device plugging in (see issue 0006491 for details).

Fixed in build 1316.

peke

2010-12-12 05:00

developer   ~0021760

Verified 1334

lowlander

2011-01-14 17:56

developer   ~0022364

User reports that MediaMonkey is warning it will delete files it shouldn't on 1343: http://www.mediamonkey.com/forum/viewtopic.php?f=6&t=54949

Ludek

2011-01-17 14:54

developer   ~0022374

Fixed in build 1344,
i.e. MediaMonkey suggests to delete only tracks in the folders configured in Device -> Options -> Destination
in order to prevent from deletion of ringtones, system event sounds, sound recorder clips, etc.

peke

2011-05-02 23:33

developer   ~0024756

Verified 1368