27 posts / 0 new / Last post
#1  January 1, 2018 - 4:12pm
danemacmillan's picture
Plex Tester
Offline
Joined: Jul 2017
Posts: 130
Kudos: 94

Infuse will currently sync corrections made in a library to iCloud, but everything else remains on each device. This means that every device is required to re-fetch all the metadata and library matching. This approach is fine for small libraries, but it's inefficient, time-consuming, and borderline impossible for large libraries.

In order for my Apple TV to index everything, it took two days of leaving it on and not watching anything, because Infuse stops fetching while streaming. In that time, I was also API banned from Google Drive for 16 hours, as my entire library is located on that service; it should be noted that when API banned, Google Drive does not let you stream anything out of it, either, so I wasn't allowed to watch anything during that time. Obviously, this is a huge deterrent in reproducing my libraries elsewhere. I simply won't do it, because I don't want to get API banned again, which I absolutely will. Multiply this by all my devices, and it's clear this is not the best approach. I also just don't want to have to keep every device open and effectively rendering them unusable during that indexing period.

I really want the ability to configure Infuse to upload the entirety of my Infuse library index and metadata to iCloud. I get the reason for only syncing corrections to iCloud right now, but there really should be an option to allow everything to sync up to iCloud for those of us who don't want to waste hours or days waiting to reproduce our large collections. Personally, a few GBs in iCloud is nothing. I would gladly pay the $1.29/mo to Apple's iCloud subscription (if I had no space left) for 50GB of storage if it meant that all my Infuse libraries, indexes, and metadata were saved. Provide a warning in the settings, so people can't accidentally enable it without knowing what they're doing, and voilà. This would mean that all of the work of getting Infuse to a usable state could be reproduced on multiple devices with very little effort, and, most importantly, very little time. I don't see any other way for Infuse to be able to scale with larger libraries.

I say everything should be synced. This means that when iCloud downloads the dump to a new device, that device will show all the libraries and all their metadata--without need to scan the underlying directory structure. If a file has been deleted since the dump, then it can be marked as unavailable. Plex handles it similarly, and the only way to remove dead files is to hit the "Empty Trash" option. We can then choose to run a new scan, but it shouldn't auto-delete these files, and only add new ones.

  January 11, 2018 - 12:36am
alizi's picture
Offline
Joined: Dec 2017
Posts: 8
Kudos: 1

+1
(I have 1 TB iCloud storage, ATV3, multiple iPads and iPhones)

  January 11, 2018 - 10:15am
ice pube's picture
Beta Tester
Offline
Joined: Jan 2018
Posts: 12
Kudos: 3

Absolutely agree. I felt the pain of trying to set up my Apple TV / iPad / iPhone. 1577 movies and 13k tv episodes has been extremely painful.

  January 11, 2018 - 11:32am
timstephens24's picture
Beta Tester
Offline
Joined: Dec 2015
Posts: 459
Kudos: 85

That would be nice. I have 200GB in iCloud, or alternatively a lot more in Gdrive (already connected to Infuse). If Infuse could set a folder there for the artwork/metadata and the paths for Google Drive that would help alleviate a lot of the issues using Google Drive.

Apple TV 4K --> Denon X4200W 7.1.2 --> LG OLED65C6P
======== SYNOLOGY DS1815+ DS513 ========
iPhone 6S Plus ||| iPad Pro 10.5

  January 11, 2018 - 1:07pm
bluskye's picture
Offline
Joined: Nov 2017
Posts: 62
Kudos: 11

How is downloading a large or small set of meta-data from iCloud for each device faster then downloading same meta-data through the through current mechanism? You still have to download something for each device.

  January 11, 2018 - 1:23pm
danemacmillan's picture
Plex Tester
Offline
Joined: Jul 2017
Posts: 130
Kudos: 94
  bluskye wrote:

How is downloading a large or small set of meta-data from iCloud for each device faster then downloading same meta-data through the through current mechanism? You still have to download something for each device.

You don't see the obvious benefit? Downloading is not the issue. Scanning is the issue. If all the hard work of scanning, which takes days for my (and other people's) libraries, could be reduced to a single download of several GBs that will take a couple minutes at most to download, the benefit is clear.

In your opinion, how is this not faster? You think it's faster to scan tens of tens of thousands files in a filesystem (or cloud service), then makes tens of thousands of requests to an API to associate the metadata to them, and to do this on every device? It's infinitely faster to download the resulting payload of all that scanning and metadata associations work, which is a single, dumb operation, versus all of that work.

_________________________________________________

  • Infuse Pro Lifetime
  January 11, 2018 - 1:26pm
bluskye's picture
Offline
Joined: Nov 2017
Posts: 62
Kudos: 11
  danemacmillan wrote:

You don't see the obvious benefit? Downloading is not the issue. Scanning is the issue. If all the hard work of scanning, which takes days for my (and other people's) libraries, could be reduced to a single download of several GBs that will take a couple minutes at most to download, the benefit is clear.

In your opinion, how is this not faster? You think it's faster to scan tens of tens of thousands files in a filesystem (or cloud service), then makes tens of thousands of requests to an API to associate the metadata to them, and to do this on every device? It's infinitely faster to download the resulting payload of all that scanning and metadata associations work, which is a single, dumb operation, versus all of that work.

It still as to scan all the files on your share regardless to ensure they didnt change, then re-sync/download meta-data. Not sure about you, but ppl delete/add files constantly and Infuse does not display meta-data for files that dont exist.

If you dont like the scanning per device, there's this client/server called Plex.

  January 11, 2018 - 1:57pm
danemacmillan's picture
Plex Tester
Offline
Joined: Jul 2017
Posts: 130
Kudos: 94
  bluskye wrote:

It still as to scan all the files on your share regardless to ensure they didnt change, then re-sync/download meta-data. Not sure about you, but ppl delete/add files constantly and Infuse does not display meta-data for files that dont exist.

If you dont like the scanning per device, there's this client/server called Plex.

You seem to misunderstand the purpose of the suggestions forum. A post made in it implies that I disagree with how something behaves or performs, and I therefore want it to behave or perform differently--preferably as described in my suggestion. My suggestion is not a misunderstanding of how it currently behaves or performs, which you appear to understand well enough. If it were to continue to behave as you describe, then that means the suggestion hasn't been implemented, which means it's still just a suggestion. If it gets implemented, then it's a new feature. You see the point I've made? You're wasting everyone's time with your pointless criticism, based on an unfortunate misunderstanding of the point of this thread. Try not to drag it out any further.

_________________________________________________

  • Infuse Pro Lifetime
  January 11, 2018 - 2:59pm
bluskye's picture
Offline
Joined: Nov 2017
Posts: 62
Kudos: 11
  danemacmillan wrote:

You seem to misunderstand the purpose of the suggestions forum. A post made in it implies that I disagree with how something behaves or performs, and I therefore want it to behave or perform differently--preferably as described in my suggestion. My suggestion is not a misunderstanding of how it currently behaves or performs, which you appear to understand well enough. If it were to continue to behave as you describe, then that means the suggestion hasn't been implemented, which means it's still just a suggestion. If it gets implemented, then it's a new feature. You see the point I've made? You're wasting everyone's time with your pointless criticism, based on an unfortunate misunderstanding of the point of this thread. Try not to drag it out any further.

"Scanning is the issue. If all the hard work of scanning, which takes days for my (and other people's) libraries, could be reduced to a single download of several GBs that will take a couple minutes at most to download, the benefit is clear."

Plex. Nuff said.

  January 11, 2018 - 4:48pm
danemacmillan's picture
Plex Tester
Offline
Joined: Jul 2017
Posts: 130
Kudos: 94
  bluskye wrote:

Plex. Nuff said.

You should start a new thread in the suggestions forum--and suggest that the Firecore team cease development of Infuse because Plex. I'm dumbstruck that I hadn't thought of it first. You beat me to it. You're right, though. There can't be two pieces of software that do what I want. Two is absurd.

_________________________________________________

  • Infuse Pro Lifetime
  January 12, 2018 - 8:45am
TheJesusFish's picture
Beta Tester
Offline
Joined: Jan 2018
Posts: 17
Kudos: 7

This would be really great. +1

  January 16, 2018 - 5:15am
danemacmillan's picture
Plex Tester
Offline
Joined: Jul 2017
Posts: 130
Kudos: 94

I just added a new device and I've been banned (temporarily) for hitting download quota (https://i.imgur.com/QY6xL3v.png), as expected. You can determine this a couple of ways (https://forum.rclone.org/t/how-to-know-if-24-hour-banned-from-google-dri...), but the easiest is to just log into the Web interface for Drive and attempt to download a large file. Once you're familiar with the issue, though, you'll know you're banned when you just can't stream anything to any of your devices anymore; you'll be greeted by various client errors. The ban typically lasts between 12 and 24 hours, but many have reported longer waits. Infuse filesystem scanning is too intense and sub-optimal. This needs to be improved. When Plex Cloud was first released they also had these problems, but eventually resolved them, as did plexdrive, a third party project. In a different ticket to improve Google Drive scanning (https://firecore.com/forum/topic/19647), it's mentioned that the solution to the problem is to look at the Drive change list. I'd be very surprised if the Infuse team told me they're using Google Drive's API to perform scans, because the performance is really on par with basic recursive directory traversal techniques, which is really not optimal when it comes to scanning cloud services, which are not filesystems: they just emulate directory structures. Based on the outward performance, it behaves like the Infuse team wrote the codebase for scanning directories, and later on, once it was figured out how to connect to Google Drive, that same codebase was pointed to the Drive. While that is okay to get things going and saves time, that will definitely not scale, and a new scanning codebase will need to be written specifically for Google Drive, and any other cloud service provider offered. Given that Google Drive has by far the most use (because it has by far the best pricing), this treatment should be prioritized for it first.

_________________________________________________

  • Infuse Pro Lifetime
  January 16, 2018 - 7:08pm
morpheus's picture
Plex Tester
Offline
Joined: Jan 2018
Posts: 23
Kudos: 9

I really hope someone at Firecore/Infuse is paying attention to your thread because we really need this.

  February 19, 2018 - 4:41pm
mcar91's picture
Offline
Joined: Feb 2018
Posts: 2
Kudos: 1

I second this request. I have multiple Apple TVs. It takes several minutes when I launch Infuse for the app to scan my library, find new files, and then assign the metadata from the cloud backend. There should be a background sync happening between TV 1 to iCloud and then iCloud to TV2 so that new files are waiting for me in my library.

Please!!!

  February 19, 2018 - 9:19pm
danemacmillan's picture
Plex Tester
Offline
Joined: Jul 2017
Posts: 130
Kudos: 94
  mcar91 wrote:

It takes several minutes[...]

I'm extremely jealous that it only takes you several minutes. Regardless, I think we can both agree that's several minutes too long!

_________________________________________________

  • Infuse Pro Lifetime
  June 11, 2018 - 5:13am
dg1974it's picture
Offline
Joined: Apr 2018
Posts: 22
Kudos: 3

+1

QNAP TVS-871-i3-4G 8x WD Red 6TB raid 6 running Plex server; Asus RT-AC88U; iPhone 8 Plus 256GB; iPad Pro 10.5" 512GB WIFI+4G; Apple TV 4K 64GB

  June 11, 2018 - 5:21am
alizi's picture
Offline
Joined: Dec 2017
Posts: 8
Kudos: 1

In general I still support this topic but I've also Plex on my server and this works only GREAT!!

  June 11, 2018 - 11:43pm
claire78's picture
Offline
Joined: Jun 2018
Posts: 24
Kudos: 1

+1 for Metadata in iCloud, with an SQL database. That wouldn’t be a big size and it would contains watched infos as well.

Other hand about iCloud sync: I have an ATV and some iPhone in my family. I don't use iCloud because on ATV use NFS connection, on iPhones use WebDav. If I turn on the iCloud in Infuse set it both connections to all devices, than everything appears double.
Unfortunatelly currently iCloud useless for me.

  June 13, 2018 - 6:29am
Arctic Monkey's picture
Offline
Joined: Nov 2014
Posts: 26
Kudos: 1

Yeh, totally agree. This feature would be extremely beneficial.

I'm also wondering why there has been no response from the Infuse Team on this matter? If there is a valid reason why this can't or won't be implemented, than it would be good to be made aware of this from an official source.

Thanks.

  June 13, 2018 - 9:09am
itimpi's picture
Moderator
Offline
Joined: Dec 2010
Posts: 1,985
Kudos: 106

The roadmap thread (at the top of this sub-form) does show that “improved iCloud sync” is a planned feature upgrade for the 5.8 release although it gives no details about what this includes.

I AM A USER - NOT A FIRECORE EMPLOYEE

  June 13, 2018 - 9:14am
Arctic Monkey's picture
Offline
Joined: Nov 2014
Posts: 26
Kudos: 1

Oh OK, excellent... i missed that... thanks

  June 13, 2018 - 12:07pm
danemacmillan's picture
Plex Tester
Offline
Joined: Jul 2017
Posts: 130
Kudos: 94
  itimpi wrote:

The roadmap thread (at the top of this sub-form) does show that “improved iCloud sync” is a planned feature upgrade for the 5.8 release although it gives no details about what this includes.

Like you wrote, this states nothing about the specifics of that planned update. I would definitely want to know the specifics, particularly related to this discussion. That planned update could have no relationship to it whatsoever.

Infuse team? Throw us a bone?

_________________________________________________

  • Infuse Pro Lifetime
  June 15, 2018 - 10:02pm
MightyMouse's picture
Offline
Joined: May 2012
Posts: 27
Kudos: 12

i will start a new thread, but i would like FIrecore to take it a step further..... i would like to store all that data on my own NAS.

  July 2, 2018 - 4:27am
Burve's picture
Offline
Joined: Jul 2018
Posts: 8
Kudos: 1

While iCloud sync might be cool, how about simpler solution with just local db. I keep all my files on my NAS, and it will be much easier to just keep some sort of local database (xml) in the same folder as videos that show what is viewed and what not. I store rest of the data there anyways (images, that are used as folder icons in Infuse), so one more file for fully offline view would be awesome.

  July 3, 2018 - 6:50am
claire78's picture
Offline
Joined: Jun 2018
Posts: 24
Kudos: 1
  Burve wrote:

While iCloud sync might be cool, how about simpler solution with just local db. I keep all my files on my NAS, and it will be much easier to just keep some sort of local database (xml) in the same folder as videos that show what is viewed and what not. I store rest of the data there anyways (images, that are used as folder icons in Infuse), so one more file for fully offline view would be awesome.

https://firecore.com/forum/topic/21100
Wink

  August 13, 2018 - 5:41am
VICO's picture
Offline
Joined: Nov 2016
Posts: 1
Kudos: 0

+1000, at least for me this is a must.

With the Nas used as metadata store, then is a dream!!!!

  August 14, 2018 - 9:03pm
danemacmillan's picture
Plex Tester
Offline
Joined: Jul 2017
Posts: 130
Kudos: 94

I'm happy to see this feature request pick up interest. With 2TB of iCloud storage, I'm practically looking for excuses to fill it.

Regardless--and to the point--I haven't used Infuse on iOS for months, simply because I don't want to get banned from GDrive for two consecutive days just to get the library indexes done. It's just too much hassle having to get every individual device up to date; not to mention, despite corrections being advertised as syncing, there are usually thousands that still need to be re-corrected after an index. I've created tickets for all of these issues.

_________________________________________________

  • Infuse Pro Lifetime