diff --git a/Readme.md b/Readme.md index 8e0219b..4618a74 100644 --- a/Readme.md +++ b/Readme.md @@ -122,7 +122,8 @@ I work tirelessly to make sure your photos are always sorted and organized so yo You don't love me yet but you will. I only do 3 things. -* Firstly I organize your existing collection of photos. + +* Firstly I organize your existing collection of photos into a customizable folder structure. * Second I help make it easy for all the photos you haven't taken yet to flow into the exact location they belong. * Third but not least I promise to do all this without a yucky propietary database that some friends of mine use. @@ -152,16 +153,15 @@ Updating EXIF of photos from the command line. I'm most helpful when I'm fully utilized to keep your photos organized. -Here's an example of a very asynchronous setup. +Here's an example of how I can create 3 geographically distributed copies of your meticulously organized photo library. + * Specify a folder in your Dropbox/Google Drive to store the organized photo library. -* Set up a Hazel rule to notify me when photos arrive in `~/Downloads` so I can import them. - * The rule waits 1 minute before processing the photo which gives you a chance to move it elsewhere if it's not something you want in the library. -* Use AirDrop to transfer files from any iPhone to your laptop. That goes to `~/Downloads` for the Hazel rule to process. - * AirDrop is fast, easy for anyone to use and once the transfer is finished your don't have to stick around. I'll move it to Dropbox/Google Drive and Dropbox/Google Drive will sync it to their servers. +* Set up a cron job to import photos in `~/Ready-To-Upload`. +* Add photos to `~/Ready-To-Upload` and wait for your cron job to trigger. * Periodically recategorize photos by fixing their location or date or by adding them to an album. * Have a Synology at home set to automatically sync down from Dropbox/Google Drive. -This setup means you can quickly get photos off your or anyone's phone and know that they'll be organized and backed up in 3 locations by the time you're ready to view them. +This setup means you can quickly get photos off your phone or dSLR and know that they'll be organized and backed up in 3 locations by the time you're ready to view or share them.

@@ -209,33 +209,60 @@ OK, so what if you don't like the folders being named `2015-07-Jul/Mountain View You can add a custom folder structure by editing your `config.ini` file. This is what I include in the sample config file. +#### Custom folder examples + +Sometimes examples are easier to understand than explainations so I'll start there. If you'd like to understand my magic I explain it in more detail below these examples. You customize your folder structure in the `Directory` section of your `config.ini`. + ``` -[Directory] -date=%Y-%m-%b -location=%city +location=%city, %state +year=%Y +full_path=%year/%location + +# 2015/Sunnyvale, California + +location=%city, %state +month=%B +year=%Y +full_path=%year/%month/%location + +# 2015/December/Sunnyvale, California + +location=%city, %state +month=%m +year=%Y +date=%year-%month full_path=%date/%location + +# 2015-12/Sunnyvale, California + +full_path=%country/%state/%city + +# US/California/Sunnyvale + ``` -There needs to be 2 levels of folders and you can construct them using the date and location. Use `full_path` to determine how the 2 levels are nested. If for some reason your config is not correct I will use the default formatting which is found in `config.ini-sample`. +#### How folder customization works -The default formatting from the above config looks like `2015-07-Jul/Mountain View`. +You can construct your folder structure using a combination of the location and dates. Under the `Directory` section of your `config.ini` file you can define placeholder names and assign each a value. For example, `date=%Y-%m` would create a date placeholder with a value of YYYY-MM which would be filled in with the date from the EXIF on the photo. -#### Customizing the date format +The placeholders can be used to define the folder structure you'd like to create. The example above happens to be the default structure and would look like `2015-07-Jul/Mountain View`. -You can use any of [the standard Python time directives](https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior) to create your ideal structure. +I have a few built-in location placeholders you can use. -* To have `201601`, use `date=%Y%m` -* For `Sunday, 01 January 2016`, use `date=%A, %d %B %Y` -* Python also has some pre-built formats. So you can get `Sun Jan 01 12:34:56 2016`, by using `%c` +* `%city` the name of the city the photo was taken. Requires geolocation data in EXIF. +* `%state` the name of the state the photo was taken. Requires geolocation data in EXIF. +* `%country` the name of the country the photo was taken. Requires geolocation data in EXIF. -#### Customizing the location format +I also have some date placeholders you can customize. You can use any of [the standard Python time directives](https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior) to customize the date format to your liking. -I use the [Open Street Maps Nominatim reverse geocoding API](http://wiki.openstreetmap.org/wiki/Nominatim#Example) provided by MapQuest. You can use `city`, `state` and `country` to construct the folder name. +* `%day` the day the photo was taken. +* `%month` the month the photo was taken. +* `%year` the year the photo was taken. -* To have `Sunnyvale`, use `location=%city` -* To have `Sunnyvale-CA`, use `location=%city-%state +In addition to my built-in and date placeholders you can combine them into a single folder name using my complex placeholders. -Sometimes a location may not have all of the values available. If your format is `%city-%state` and `city` was not returned then the folder name will be `%state`. Take note that I'll strip out extra characters so you don't end up with folders name `-%state` when `city` is not found. +* `%location` can be used to combine multiple values of `%city`, `%state` and `%country`. For example, `location=%city, %state` would result in folder names like `Sunnyvale, California`. +* `%date` can be used to combine multiple values from [the standard Python time directives](https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior). For example, `date=%Y-%m` would result in folder names like `2015-12`. ### Reorganize by changing location and dates diff --git a/elodie/compatability.py b/elodie/compatability.py index 9d65f75..e0d8576 100644 --- a/elodie/compatability.py +++ b/elodie/compatability.py @@ -26,8 +26,15 @@ def _decode(string, encoding=sys.getfilesystemencoding()): def _copyfile(src, dst): - # Python 3 hangs using open/write method + # shutil.copy seems slow, changing to streaming according to + # http://stackoverflow.com/questions/22078621/python-how-to-copy-files-fast # noqa + # Python 3 hangs using open/write method so we proceed with shutil.copy + # and only perform the optimized write for Python 2. if (constants.python_version == 3): + # Do not use copy2(), it will have an issue when copying to a + # network/mounted drive. + # Using copy and manual set_date_from_filename gets the job done. + # The calling function is responsible for setting the time. shutil.copy(src, dst) return diff --git a/elodie/filesystem.py b/elodie/filesystem.py index fcbff06..940e847 100644 --- a/elodie/filesystem.py +++ b/elodie/filesystem.py @@ -114,7 +114,7 @@ class FileSystem(object): # First we check if we have metadata['original_name']. # We have to do this for backwards compatibility because # we original did not store this back into EXIF. - if(metadata['original_name'] is not None): + if('original_name' in metadata and metadata['original_name']): base_name = os.path.splitext(metadata['original_name'])[0] else: # If the file has EXIF title we use that in the file name @@ -163,18 +163,18 @@ class FileSystem(object): config_directory = config['Directory'] - path_parts = re.search( - '\%([^/]+)\/\%([^/]+)', + # Find all subpatterns of full_path that map to directories. + # I.e. %foo/%bar => ['foo', 'bar'] + path_parts = re.findall( + '\%([a-z]+)', config_directory['full_path'] ) - if not path_parts or len(path_parts.groups()) != 2: + if not path_parts or len(path_parts) == 0: return self.default_folder_path_definition - path_part_groups = path_parts.groups() self.cached_folder_path_definition = [ - (path_part_groups[0], config_directory[path_part_groups[0]]), - (path_part_groups[1], config_directory[path_part_groups[1]]), + (part, config_directory[part]) for part in path_parts ] return self.cached_folder_path_definition @@ -188,25 +188,21 @@ class FileSystem(object): path = [] for path_part in path_parts: part, mask = path_part - if part == 'date': + if part in ('date', 'day', 'month', 'year'): path.append(time.strftime(mask, metadata['date_taken'])) - elif part == 'location': - if( - metadata['latitude'] is not None and - metadata['longitude'] is not None - ): - place_name = geolocation.place_name( - metadata['latitude'], - metadata['longitude'] - ) - if(place_name is not None): - location_parts = re.findall('(%[^%]+)', mask) - parsed_folder_name = self.parse_mask_for_location( - mask, - location_parts, - place_name, - ) - path.append(parsed_folder_name) + elif part in ('location', 'city', 'state', 'country'): + place_name = geolocation.place_name( + metadata['latitude'], + metadata['longitude'] + ) + + location_parts = re.findall('(%[^%]+)', mask) + parsed_folder_name = self.parse_mask_for_location( + mask, + location_parts, + place_name, + ) + path.append(parsed_folder_name) # For now we always make the leaf folder an album if it's in the EXIF. # This is to preserve backwards compatability until we figure out how @@ -217,11 +213,6 @@ class FileSystem(object): elif(len(path) == 2): path[1] = metadata['album'] - # if we don't have a 2nd level directory we use 'Unknown Location' - if(len(path) < 2): - path.append('Unknown Location') - - # return '/'.join(path[::-1]) return os.path.join(*path) def parse_mask_for_location(self, mask, location_parts, place_name): @@ -338,11 +329,6 @@ class FileSystem(object): shutil.move(_file, dest_path) os.utime(dest_path, (stat.st_atime, stat.st_mtime)) else: - # Do not use copy2(), will have an issue when copying to a - # network/mounted drive using copy and manual - # set_date_from_filename gets the job done - # shutil.copy seems slow, changing to streaming according to - # http://stackoverflow.com/questions/22078621/python-how-to-copy-files-fast # noqa compatability._copyfile(_file, dest_path) self.set_utime(media) @@ -352,7 +338,7 @@ class FileSystem(object): return dest_path def set_utime(self, media): - """ Set the modification time on the file base on the file name. + """ Set the modification time on the file based on the file name. """ # Initialize date taken to what's returned from the metadata function. diff --git a/elodie/geolocation.py b/elodie/geolocation.py index 1f63718..7a6bc73 100644 --- a/elodie/geolocation.py +++ b/elodie/geolocation.py @@ -19,6 +19,7 @@ from elodie import log from elodie.localstorage import Db __KEY__ = None +__DEFAULT_LOCATION__ = 'Unknown Location' def coordinates_by_name(name): @@ -115,10 +116,14 @@ def get_key(): def place_name(lat, lon): + lookup_place_name_default = {'default': __DEFAULT_LOCATION__} + if(lat is None or lon is None): + return lookup_place_name_default + # Convert lat/lon to floats - if not isinstance(lat, float): + if(not isinstance(lat, float)): lat = float(lat) - if not isinstance(lon, float): + if(not isinstance(lon, float)): lon = float(lon) # Try to get cached location first @@ -132,19 +137,18 @@ def place_name(lat, lon): lookup_place_name = {} geolocation_info = lookup(lat=lat, lon=lon) - if(geolocation_info is not None): - if('address' in geolocation_info): - address = geolocation_info['address'] - for loc in ['city', 'state', 'country']: - if(loc in address): - lookup_place_name[loc] = address[loc] - # In many cases the desired key is not available so we - # set the most specific as the default. - if('default' not in lookup_place_name): - lookup_place_name['default'] = address[loc] + if(geolocation_info is not None and 'address' in geolocation_info): + address = geolocation_info['address'] + for loc in ['city', 'state', 'country']: + if(loc in address): + lookup_place_name[loc] = address[loc] + # In many cases the desired key is not available so we + # set the most specific as the default. + if('default' not in lookup_place_name): + lookup_place_name['default'] = address[loc] if('default' not in lookup_place_name): - lookup_place_name = {'default': 'Unknown Location'} + lookup_place_name = lookup_place_name_default if(lookup_place_name is not {}): db.add_location(lat, lon, lookup_place_name) diff --git a/elodie/tests/filesystem_test.py b/elodie/tests/filesystem_test.py index 93d66fa..336fef7 100644 --- a/elodie/tests/filesystem_test.py +++ b/elodie/tests/filesystem_test.py @@ -248,6 +248,48 @@ full_path=%date/%location assert path == os.path.join('2015-12-05','United States of America-California-Sunnyvale'), path +@mock.patch('elodie.config.config_file', '%s/config.ini-location-date' % gettempdir()) +def test_get_folder_path_with_with_more_than_two_levels(): + with open('%s/config.ini-location-date' % gettempdir(), 'w') as f: + f.write(""" +[Directory] +year=%Y +month=%m +location=%city, %state +full_path=%year/%month/%location + """) + + if hasattr(load_config, 'config'): + del load_config.config + + filesystem = FileSystem() + media = Photo(helper.get_file('with-location.jpg')) + path = filesystem.get_folder_path(media.get_metadata()) + if hasattr(load_config, 'config'): + del load_config.config + + assert path == os.path.join('2015','12','Sunnyvale, California'), path + +@mock.patch('elodie.config.config_file', '%s/config.ini-location-date' % gettempdir()) +def test_get_folder_path_with_with_only_one_level(): + with open('%s/config.ini-location-date' % gettempdir(), 'w') as f: + f.write(""" +[Directory] +year=%Y +full_path=%year + """) + + if hasattr(load_config, 'config'): + del load_config.config + + filesystem = FileSystem() + media = Photo(helper.get_file('plain.jpg')) + path = filesystem.get_folder_path(media.get_metadata()) + if hasattr(load_config, 'config'): + del load_config.config + + assert path == os.path.join('2015'), path + def test_get_folder_path_with_location_and_title(): filesystem = FileSystem() media = Photo(helper.get_file('with-location-and-title.jpg')) @@ -658,3 +700,47 @@ full_path=%date/%location ] if hasattr(load_config, 'config'): del load_config.config + +@mock.patch('elodie.config.config_file', '%s/config.ini-location-date' % gettempdir()) +def test_get_folder_path_definition_with_more_than_two_levels(): + with open('%s/config.ini-location-date' % gettempdir(), 'w') as f: + f.write(""" +[Directory] +year=%Y +month=%m +day=%d +full_path=%year/%month/%day + """) + + if hasattr(load_config, 'config'): + del load_config.config + filesystem = FileSystem() + path_definition = filesystem.get_folder_path_definition() + expected = [ + ('year', '%Y'), ('month', '%m'), ('day', '%d') + ] + if hasattr(load_config, 'config'): + del load_config.config + + assert path_definition == expected, path_definition + +@mock.patch('elodie.config.config_file', '%s/config.ini-location-date' % gettempdir()) +def test_get_folder_path_definition_with_only_one_level(): + with open('%s/config.ini-location-date' % gettempdir(), 'w') as f: + f.write(""" +[Directory] +year=%Y +full_path=%year + """) + + if hasattr(load_config, 'config'): + del load_config.config + filesystem = FileSystem() + path_definition = filesystem.get_folder_path_definition() + expected = [ + ('year', '%Y') + ] + if hasattr(load_config, 'config'): + del load_config.config + + assert path_definition == expected, path_definition